A video decoder is provided that includes memory and a processor coupled to the memory. The processor may be configured to convert a bitstream into inter-prediction parameters and reconstruct motion data based on the inter-prediction parameters. The processor may further be configured to refine the motion data based on finding a match between a current template of a current picture and a reference template of a reference picture and perform a motion compensation operation with the refined motion data and a reference block to generate an inter-prediction block. The processor may be configured to add the inter-prediction block to an inter-residual block to produce a reconstructed block. The motion data may be reconstructed without refined motion data associated with a previous prediction unit.