A system of detecting a motion vector for an object block in a current frame of an orthogonally transformed video with respect to a stored reference frame. The motion vector is detected by: extracting a predetermined pattern of lower frequency components from the object block; orthogonally inverse transforming the extracted lower frequency components within the pattern to yield the pattern of pixel data; searching the reference frame for a block having a maximum correlation with the pattern of pixel data; and calculating the motion vector from the object block to the block having a maximum correlation. As an orthogonal transform, DCT (discrete cosine transform) is preferable. Creating a row and a column image comprising sums obtained by adding all the pixels in each column and each row of the object block and applying the above searching and calculating process to the row and the column image enables the motion vector detection from a normal or noncompressed video.