A multi-layered video encoding method is provided wherein motion estimation is performed by using one of two frames of a lower layer temporally closest to an unsynchronized frame of a current layer as a reference frame. A virtual base layer frame at the same temporal location as that of the unsynchronized frame is generated using a motion vector obtained as a result of the motion estimation and the reference frame. The generated virtual base layer frame is subtracted from the unsynchronized frame to generate a difference, and the difference is encoded.