An apparatus and concomitant method for creating and applying an encoded bitstream to evaluate the performance of a black box video decoder. The encoded bitstreams are designed such that the results from the DUT can be evaluated without the need of a human observer. The method applies the encoded bitstream to both a DUT and a compliant reference decoder and compares the results from both decoders to determine compliance of the DUT. Additional embodiments address postprocessing and the selection of a subset of test frames for evaluation.