A video is analyzed to identify interest level for a plurality of frames. The interest level is determined based on analysis of the content of each of the plurality of frames or how often other clients view the frames or take actions while viewing the frames. Frames are clustered and key frame locations are determined for each of the clusters. Key frames are given a higher buffering priority than non key frames. Buffering of key frames is prioritized based on key frame interest level. Key frames are also utilized to enable a client to skip forward or backward between key frames and view content that is likely to be of interest to the user.