A video processing system includes a video processing unit (VPU) and one or more display processing units, all having access to external memory. Video data representing frames to be displayed is generated. The VPU generates pixel data representing the frames and stores it in memory. The display processing units then read the pixel data to display the frames. The VPU is configured to generate and store in memory pixel data representing reference frames for the sequence of video frames at the full resolution of the reference frame and also at at least one lower resolution to the full resolution.