Disclosed herein are systems, methods, and non-transitory computer-readable storage media for presenting a contact manager associated with a communication session represented in a graphical user interface (GUI). The method includes grouping by context contacts associated with a user in the communication session, wherein the communication session is displayed as a set of connected graphical elements representing a structure of the communication session, presenting on the GUI a graphical representation of at least a part of the grouped contacts, receiving user input associated with one contact of the contacts and the communication session, and modifying the communication session based on information associated with the user input and the one contact. Grouping the contacts by context can include sorting the contacts as a cache, and grouping based on user preferences, user communications history, social networks, topic, project, and/or employer.