A method for indexing a sports video program carried by a video stream is disclosed. The method includes temporarily indexing a plurality of shots of the video stream by performing shot detection. In addition, the method further includes extracting a plurality of shots of the sports video program from at least one portion of the shots indexed during the shot detection at least according to low level visual features of the at least one portion, whereby the extracted shots are temporarily kept indexed. Additionally, the method further includes validating a plurality of shots by comparing one of the extracted shots with at least the others of the extracted shots, whereby the validated shots are kept indexed for use of playback of at least one portion of the sports video program.