The disclosure is related to a method and a system for removing background of a video. One of the objectives of the method and system is to extract a motion image object from a video. In the method, a plurality of frames is first retrieved from the video. An image processing approach is performed to recognize image features of every frame. A plurality of color blocks for every frame are obtained through the image processing scheme. A variance among the pixels in every color block of two continuous frames can be obtained by a comparison computation, so as to form a color block variance. The color block variance allows the system to determine a motion target object (211') as well as its image contour. The blocks excepting the blocks of the target object (211') are set as transparent blocks. Therefore the background relative to the target object (211') can be removed.