A method of training a cognitive system comprises identifying, by a cognitive training device, a performance specification associated with a cognitive device; determining, by the cognitive training device, a forecasted number of communication inputs to be received by the cognitive device to meet the performance specification; identifying, by the cognitive training device, each communication input received by the cognitive device; and when the number of communication inputs reaches the forecasted number of communication inputs, automating, by the cognitive training device, identification of intents associated with additional communication inputs received by the cognitive device. Additionally, the cognitive training device can include a statistical model employed to determine the random sampling method of the experts utilized during training of the cognitive system in order to classify the gathered input into appropriate intents while mitigating incorporation of bias.