An approach is provided in which an information handling system analyzes correction information corresponding to an answer generated by a question answer system. The correction information includes a correction to the answer and captured variable information utilized by the question answer system to generate the answer. The information handling system selects input variables based upon the correction analysis and generates a test case that includes the selected input variables and the variable information. In turn, the information handling system tests the question answer system using the generated test case.