Embodiments described herein provide a method of evaluating a natural language processing model. The method includes receiving an evaluation dataset that may include a plurality of unit tests, the unit tests having: an input context, and a first candidate and a second candidate that are generated in response to the input context, where the first test candidate is associated with a first quality notation, and the second candidate is associated with a second quality notation. The method includes determining, via a model, a first likelihood of generating the first candidate and a second likelihood of generating the second candidate in response to the input context. The method also includes determining whether the first likelihood being greater than the second likelihood. The method also includes determining whether the first model passed the unit test, where the first quality notation indicates a higher quality candidate and the second quality notation indicate a lower quality candidate.