A method for generating reliability tests for a telephone system is based upon sampling an orthogonal array which covers various combinations of test parameters. Field data is collected of actual telephone activity on a telephone system. The field data is evaluated so as to determine call-mix characteristics. Probabilistic weights for the different call-mix characteristics are obtained, and then the probabilistic weights are used to sample the test case scenarios generated in the orthogonal array which have the same call-mix characteristics. These test case scenarios are used to run tests on the telephone system. These tests are preferably performed using automated test scripts. After the test data is collected, reliability metrics are calculated from the test data.