An example device for decoding video data includes a memory configured to store video data, and a video decoder implemented in circuitry and configured to determine that motion information of a current block of the video data is to be derived using decoder-side motion vector derivation (DMVD), determine a pixels clue for the current block, the pixels clue comprising pixel data obtained from one or more groups of previously decoded pixels, derive the motion information for the current block according to DMVD from the pixels clue, and decode the current block using the motion information. The video decoder may generate the pixels clue using multiple hypothesis predictions from multiple motion compensated blocks. The video decoder may determine an inter-prediction direction for the motion information according to matching costs between different prediction directions. The video decoder may refine the motion information using a calculated matching cost for the pixels clue.