A system is provided where a number of representative video snapshots are displayed on a timeline indicating a position to jump to in a video using control buttons, such as fast forward, reverse, or a timeline scroll bar. The video snapshots are obtained by opening a low resolution connection to a video server to receive forward looking video segments. The forward looking video segments are summarized and the representative video snapshots are selected and displayed to provide a reference frame for the user to select a timeline position, or snapshots directly linked to positions in the higher resolution video. Backward looking video segments may be similarly summarized from the low resolution connection, or alternatively from video previously played. As the video progresses, new forward looking video segments are received and new representative video snapshots are summarized and displayed in close proximity with the video being played.