Implementations of the teachings herein include coding video data with an alternate reference frame generated using a temporal filter. The alternate reference frame is generated by determining a first weighting factor, for each corresponding block of a respective frame of a filter set, that represents a temporal correlation of the block with the corresponding block, determining a second weighting factor, for each pixel for each corresponding block of the respective frame of the filter set, that represents a temporal correlation of the pixel to a spatially-correspondent pixel in the block, determining a filter weight for each pixel in the block and for each spatially-correspondent pixel is each corresponding block based on the first weighting factor and the second weighting factor, and generating a weighted average pixel value for each pixel position in the block to form a block of the alternate reference frame based on the filter weights.