Summaries of media programs that are in progress are generated based on content of the media programs that has already been transmitted to listeners or viewers. The content is transcribed into text, and contextual features regarding the media program such as topics, identities of speakers or interactions received from listeners are identified. The transcribed content and the contextual features are provided as multi-modal inputs to a model that is trained to generate a summary of the media program in response to such inputs. Summaries of media programs that are then in progress are transmitted to devices of listeners who may be interested in joining one of the media programs and displayed in a menu or user interface or announced to the listeners.