A system and method for retargeting video sequences are provided. A method for retargeting a video includes a plurality of frames includes determining saliency information for the plurality of frames, determining a cost metric for the video, and retargeting the video based on the cost metric to produce a retargeted video. The cost metric considers loss due to cropping, scaling, temporal factors, and spatial factors. The retargeting makes use of a crop window for each frame in the plurality of frames.