A video coding system includes a prediction module configured to perform multi-reference picture sub-pixel motion estimation. Reference pictures are selected for the sub-pixel motion estimation based on full-pixel motion estimation. A cost function is used to select the reference pictures from one or more groups of reference pictures for a block in a macroblock. The cost is based on full-pixel motion estimation errors and a predicted motion vector for the block. The selected reference pictures are used for sub-pixel motion estimation to get the final motion vector for each block.