The present invention is embodied in a system and method for maintaining a background model of an image sequence by processing on multiple spatial scales. These multiple spatial scales include a pixel scale, a regional scale and a frame scale. The image sequence undergoes pixel processing that determines a current background model and provides an initial pixel assignment as either a background or a foreground pixel. Region processing further refines the initial pixel assignments by considering relationships between pixels and possibly reassigning pixels. Frame processing further refines the current background model by determining whether a substantial change has occurred in the actual background and, if so, providing a more accurate background model.