Generally, this disclosure provides methods and systems for video indexing systems with viewer reaction estimation based on visual cue detection. The method may include detecting visual cues generated by a user, the visual cues generated in response to the user viewing the video; mapping the visual cues to an emotion space associated with the user; estimating emotion events of the user based on the mapping; and indexing the video with metadata, the metadata comprising the estimated emotion events and timing data associated with the estimated emotion events. The method may further include summarization, partitioning and searching of videos based on the video index.