In various embodiments, a recording and playback manager of a digital video recorder (DVR), set top box, receiver or other device stores and accesses statistics in a statistics file related to average distances between reference frames (e.g., intra frames or I-frames) within blocks of the video data of the recording. Accessing these statistics during playback enables the system to more accurately locate reference frames to display while executing the playback in a trick mode such as fast forward, rewind or skip. The average distance between reference frames for a particular block of video data may be used to jump from one reference frame to the next reference frame in that block during playback in trick mode.