Methods, systems, and media for summarizing a video with video thumbnails are provided. In some embodiments, the method comprises: receiving a plurality of video frames corresponding to the video and associated information associated with each of the plurality of video frames; extracting, for each of the plurality of video frames, a plurality of features; generating candidate clips that each includes at least a portion of the received video frames based on the extracted plurality of features and the associated information; calculating, for each candidate clip, a clip score based on the extracted plurality of features from the video frames associated with the candidate clip; calculating, between adjacent candidate clips, a transition score based at least in part on a comparison of video frame features between frames from the adjacent candidate clips; selecting a subset of the candidate clips based at least in part on the clip score and the transition score associated with each of the candidate clips; and automatically generating an animated video thumbnail corresponding to the video that includes a plurality of video frames selected from each of the subset of candidate clips.