A method and system of video object segmentation are disclosed herein. A pixel data of an image is received, wherein the pixel data has a corresponding location. A difference value between a pixel value of the pixel data and a pixel value in the corresponding location of a previous image is obtained. Utilizing the difference value and the pixel data, a multi-layered background registration is performed to get a background. Using the background, a background region of the image is removed. Furthermore, a process of removing an average value of luminance from the image and from the background is carried out to prevent object segmentation failure caused by the non-uniform luminance problem, which is produced by variation and flickering of illumination.