In one example, a device includes a memory configured to store the video data; and one or more processors implemented in circuitry and configured to determine a plurality of distances between a first representative point of a current block of video data and a plurality of second representative points of neighboring blocks to the current block, add one or more of the neighboring blocks as candidates to a candidate list of the current block in an order according to the distances between the first representative point and the second representative points, and code the current block using the candidate list. The candidate list may be, for example, a merge list, an AMVP list, or a most probable mode list. Alternatively, the candidate list may be a list of candidates from which to determine context information for context-adaptive binary arithmetic coding (CABAC).