Disclosed are various embodiments for segmenting and classifying video content using sounds. In one embodiment, a plurality of segments of a video content item are generated by analyzing audio accompanying the video content item. A subset of the plurality of segments that correspond to music segments is selected based at least in part on an audio characteristic of the subset of the plurality of segments. Individual segments of the subset of the plurality of segments are processed to determine whether a classification applies to the individual segments. A list of segments of the video content item to which the classification applies is generated.