A method for processing video image is provided to perform a motion vector filtering process on the image after being processed by a motion estimation process. The motion vector filtering process adjusts the motion vectors derived for each block in a video sequence, so that the motion vector of each block and the actual moving direction of the corresponding object in the image tend to conformity. Then, the blocks with similar motion vectors are merged and blocks with radical motion vectors are corrected, thereby reducing distortion of the output image and the number of bits used to represent the motion information, which in turns increases the efficiency of compression process.