A method and apparatus of deriving a motion vector predictor (MVP) for a current block in an Inter, Merge, or Skip mode are disclosed. Embodiments according to the present invention determine redundant MVP candidates according to a non-MV-value based criterion. The redundant MVP candidates are then removed from the MVP candidate set. In other embodiments according to the present invention, motion IDs are assigned to MVP candidates to follow the trail of motion vectors associated with the MVP candidate. An MVP candidate having a same motion ID as a previous MVP is redundant and can be removed from the MVP candidate set. In yet another embodiment, redundant MVP candidates correspond to one or more of the MVP candidates that cause the second 2N×N or N×2N PU to be merged into a 2N×2N PU are removed from the MVP candidate set.