A method of coding video data includes determining a candidate motion vector for each of one or more candidate portions of a video frame and determining a current motion vector for a current portion of a current frame. The current motion vector identifies a portion of a reference frame that at least partially matches the current portion of the current frame. The method also includes calculating a motion vector difference between the current motion vector and each of the candidate motion vectors, selecting one of the candidate motion vectors based on the calculated motion vector differences, signaling an index identifying the candidate portion having the selected one of the candidate motion vectors, and signaling the corresponding motion vector difference calculated with respect to the selected one of the candidate motion vectors.