Video information defining video content may be accessed. Highlight moments within the video content may be identified. Flexible video segments may be determined based on the highlight moments. Individual flexible video segments may include one or more of the highlight moments and a flexible portion of the video content. The flexible portion of the video content may be characterized by a minimum segment duration, a target segment duration, and a maximum segment duration. A duration allocated to the video content may be determined. One or more of the flexible video segments may be selected based on the duration and one or more of the minimum segment duration, the target segment duration, and/or the maximum segment duration of the selected flexible video segments. A video summary including the selected flexible video segments may be generated.