A method and system are provided in which a video image may be scaled from a first to a second video format. The scaling may be a non-uniform scaling such as an anamorphic scaling. When panning associated with the scaled video image is detected, one or more end portions of a current frame of the scaled video image may be adjusted, the adjustment being based on one or more frames of the scaled video image that are previous to the current frame. The adjustment may result from combining information from the corresponding end portion of previous frames with the information of the current frame. One or more end portions that are opposite to the ones adjusted may also be adjusted based on one or more frames of the scaled video image that are subsequent to the current frame. The panning detected may be horizontal, vertical, or a combination thereof.