In general, embodiments of the present invention provide systems, methods and computer readable media for a configurable test environment within which a relevance service can be invoked to execute one or a combination of test scenarios, each test scenario respectively being configured to exercise one or a combination of features of the relevance service. In embodiments, a test scenario may be configured to use test data that can be simulated and/or be derived from one or a combination of user models and promotion models, and/or be based on aggregated data that has been collected from previous production runs of the relevance service. In embodiments, each test scenario is described as a set of test configuration data. In some embodiments, the test configuration data are represented in a data interchange format that is both human and machine-readable, e.g., JavaScript Object Notation (JSON).