A method and apparatus of improving the compression efficiency of a motion vector is provided. The method includes obtaining a motion vector in a base layer frame having a first frame rate from an input frame, obtaining a motion vector in a first enhancement layer frame having a second frame rate from the input frame, the second frame rate being greater than the first frame rate, generating a predicted motion vector by referring to a motion vector for at least one frame among base layer frames present immediately before and after the same temporal position as the first enhancement layer frame if there is no base layer frame at the same temporal position as the first enhancement layer frame, and coding a difference between the motion vector in the first enhancement layer frame and the generated predicted motion vector, and the obtained motion vector in the base layer.