Disclosed herein are systems, methods, and non-transitory computer-readable storage media for generating a context-based rich communication log. A system configured to practice the method performs a context-based analysis of a history of communication sessions over at least two communication modalities, wherein the context-based analysis considers semantic meaning, temporal relationship, and user activities. The system generates associations between the communication sessions based on the context-based analysis, and generates the context-based rich communication log based on the associations. The context-based analysis can be based on time, session participant identifications, ongoing events, and/or on-demand queries of the communication sessions. The system can then present the context-based rich communication log to a user. The context-based rich communication log can be used as part of at least one of a predictive contacts application, a topic analyzer, a conferencing application, and a personal communication assistant, for example.