A method for identifying a set of key frames from a video sequence including a time sequence of video frames, comprising: extracting a feature vector for each video frame in a set of video frames selected from the video sequence; defining a set of basis functions that can be used to represent the extracted feature vectors, wherein each basis function is associated with a different video frame in the set of video frames; representing the feature vectors for each video frame in the set of video frames as a sparse combination of the basis functions associated with the other video frames; and analyzing the sparse combinations of the basis functions for the set of video frames to select the set of key frames.