The invention provides a motion prediction method. First, a plurality of candidate units corresponding to a current unit of a current frame is determined. A plurality of motion vectors of the candidate units is then obtained. A plurality of scaling factors of the candidate units is then calculated according to a plurality of respective temporal distances depending on a plurality of reference frames of the motion vectors. The motion vectors of the candidate units are then scaled according to the scaling factors to obtain a plurality of scaled motion vectors. The scaled motion vectors are ranked, and a subset of highest ranking motion vectors are identified to be included in a candidate set. Finally, a motion vector predictor for motion prediction of the current unit is then selected from the candidate units.