A skip behavior analyzer is part of a media delivery system that allows for unbiased A/B testing of a plurality of versions of a song. The media delivery system stores a plurality of versions of a song and randomly selects, for each requesting device, a version of the song to associate with that device. Each time the device requests the song, thereafter, the media delivery system will provide the same version of the song for consistency. The media delivery system then gathers song play and skip information, calculates the differences in distribution of the skip behavior, and provides the skip information to allow a music composer to better determine which version of a song is more popular and why that is so.