Methods and apparatus for determining highlights from broadcasts in spectating environments. A highlight service obtains highlight data for broadcasts including game- and participant-specified events, audio input, and text chat, analyzes the highlight data to determine notable events (highlights) in the broadcasts, and extracts highlight segments from the broadcasts according to the determined events. Highlight reels may be created from the highlight segments according to one or more highlight selection criteria and/or spectator preferences. The highlight service may provide access to the highlights and highlight reels via a highlight user interface (UI). The spectators may selectively view the highlights or highlight reels via the highlight UI.