Systems and methods for generating and using performance metadata associated with a media piece (e.g., music or video) are provided. An electronic device can monitor a user's workout performance while playing a particular media piece. Workout performance may include the user's rate of motion and/or one or more physiological metrics of the user. Based on the user's workout performance, the electronic device can create new or modify existing performance metadata associated with the media piece. In some embodiments, the performance metadata based on a particular user's workout performance in response to a media piece may be combined with collective performance metadata based on the workout performances of multiple users in response to the media piece. The combined performance metadata may then be stored as new collective metadata. Accordingly, the collective performance metadata can represent the average response to a media piece over multiple users.