Certain embodiments involve generating summarized versions of video content based on memorability of the video content. For example, a video summarization system accesses segments of an input video. The video summarization system identifies memorability scores for the respective segments. The video summarization system selects a subset of segments from the segments based on each computed memorability score in the subset having a threshold memorability score. The video summarization system generates visual summary content from the subset of the segments.