An email system is disclosed which provides annotations associated with email to recipients and allows the recipients to interact with annotated email. The system receives email and determines whether the received email includes items that should be annotated. Responsive to determining that an email includes items that should be annotated, the system acquires annotated information related to the email. The received email and associated annotated information are stored in the email system. When a request is received to access email for a specific user, the email system displays an email listing, identifying email that is associated with an annotation using a visual representation. If the recipient responds to an annotation, the system receives and stores the recipient's response. The system then sends the recipient's response to the sender of the email. The system also receives updates to received email from email senders and displays email updates to the appropriate recipients.