Provided are a method of generating a trained third neural network to recognize a speaker of a noisy speech signal by combining a trained first neural network which is a skip connection-based neural network for removing noise from the noisy speech signal with a trained second neural network for recognizing the speaker of a speech signal, and a neural network device for operating the neural networks.