Systems and methods for classifying messages are provided. Each message in a plurality of messages is classified, thereby independently identifying a message category in a set of message categories for each respective message in the plurality of messages. The plurality of messages is delivered to a plurality of recipients with a designation of the message category of each respective message in the first plurality of messages. A plurality of recipient initiated message interaction events for messages in the first plurality of messages over a predetermined period of time is collected from the plurality of recipients. A message categorization dataset is then constructed from (i) the first plurality of messages, (ii) the designation of the message category of each respective message in the subset of the first plurality of messages, and (iii) the plurality of recipient initiated message interaction events. This message categorization dataset is used to train or evaluate a message classifier.