A method for controlling a video segmentation apparatus is provided. The method includes receiving an image corresponding to a frame of a video; estimating a motion of an object in the received image to be extracted from the received image, determining a plurality of positions of windows corresponding to the object; adjusting at least one of a size and a spacing of at least one window located at a position of the plurality of determined positions of the windows based on an image characteristic; and extracting the object from the received image based on the at least one window of which the at least one of the size and the spacing is adjusted.