A method for processing a video includes maintaining one or more tables, wherein each table includes motion candidates derived from previously processed video blocks that are processed prior to a first video block of a video; deriving motion information for the first video block; applying refinement to the motion information when at least one decoder side motion vector derivation (DMVD) technologies is enabled; performing a conversion between a bitstream representation of the first video block and the first video block using the refined motion information; and determining, whether to use the refined motion information or un-refined motion information for updating one or more tables.