A quality metric of a multimedia summary of a multimedia content item is determined based, in part, on semantic similarities of the summary and content item, rather than just on word frequencies. This is accomplished in some embodiments by identifying a semantic meaning of the summary and multimedia content item using vector analysis. The vectors of the summary and the vectors of the multimedia content item are compared to determine semantic similarity. In other examples, the quality metric of the multimedia summary is determined based on, in part, a coherence between an image portion of a summary and a text portion of the summary for determining a quality metric of a multimedia summary.