A method and apparatus of deriving a temporal derived motion vector in a second direction based on a given motion vector in a first direction for motion vector prediction are disclosed. According to the present invention, a given motion vector for a current block is determined, where the given motion vector points from the current block in a first direction. A reference motion vector associated with a first reference block in a first reference frame is identified. Then, based on the reference motion vector and the given motion vector, a temporal derived motion vector is derived. The temporal derived motion vector points from the current block to a second reference block in a second reference frame in a second direction different from the first direction. The temporal derived motion vector is then used as one predictor for encoding or decoding of the motion vector of the current block.