A set of variations of a user experience on a subscription-based online site are tested, where each of the set of variations is distinct from other variations in the set of variations. Each of a group of users is assigned to one of the set of variations. For each user request received during a predetermined window of time, one of the set of variations is determined for use in processing the user request, based on a user associated with the user request, and the respective user request is processed using the determined variation. Long-term user metric information relating to subscription services for the subscription-based online site is determined, with respect to the group of users. At least one of the set of variations is then rated based on the determined long-term user metric information.