A method, system, and computer-readable storage medium are disclosed for upscaling an image sequence. An upsampled frame is generated based on an original frame in an original image sequence comprising a plurality of frames. A smoothed image sequence is generated based on the original image sequence. A plurality of patches are determined in the upsampled frame. Each patch comprises a subset of image data in the upsampled frame. Locations of a plurality of corresponding patches are determined in a neighboring set of the plurality of frames in the smoothed image sequence. A plurality of high-frequency patches are generated. Each high-frequency patch is based on image data at the locations of the corresponding patches in the original image sequence. The plurality of high-frequency patches are added to the upsampled frame to generate a high-quality upscaled frame.