Automatically marking locations of user interest in media content is disclosed. Data that describes user interaction with media content is analyzed to determine locations of potential interest. For each of the locations of potential interest, the data is analyzed to determine whether user interest satisfies a criterion for establishing a mark for the location. Marks are determined for those locations that satisfy the criterion. A mark includes a description for a given location and information that allows the given location to be automatically presented to a user upon selection of the description. In another aspect, extrinsic data that is separate from a piece of media content is analyzed to determine whether moments of high user interest might exist in the piece of media content. Then, the piece of media content is scanned to find locations in the media content that correspond to the moments.