A video surveillance system is disclosed which readily distinguishes between changes within an observed scene, caused by the presence of an intruder within the scene, and changes resulting from lighting variations. The video surveillance system implements a retinex-based algorithm for distinguishing the presence of a new object within the scene from changes in scene lighting. Areas of change between a current image (F1) of the scene and a reference image (RF) thereof are identified to produce a difference image. Response to significant differences present in the difference image, the entire current image and a subsequent image (F2) are partitioned into uniform segments (SF1.sub.1 -SF1.sub.n, SF2.sub.1 -SF2.sub.n), and ratios of changes in light intensity along the edges of the uniform segments are calculated for each partition. If a comparison of the resultant edge ratios between corresponding segments in the images indicates a change in the scene, the process is repeated for at least one subsequent image frame. If similar results are attained, the system signals an alarm condition to a monitoring station or a system operator.