Methods and apparatus, including computer program products, implementing and using techniques for evaluating indexing systems. Comparative tests are performed on two or more indexing systems. Each test uses one or more of: benchmarks, metrics, estimates of input data, desired index configuration, and performance constraints. A recommendation is automatically provided for an indexing system and an optimal configuration of the indexing system is also provided, based on an evaluation metrics from the tests.