Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a neural network. One of the methods includes accessing a first neural network that was trained to recognize a given keyword or keyphrase using a set of hotword training data, wherein the hotword training data includes positive hotword training data that correspond to utterances of the keyword or keyphrase, and negative hotword training data that corresponds to utterances of words or phrases that are other than the keyword or keyphrase, selecting a seed hotsound, mapping, to a feature space, (i) the positive hotword training data, (ii) the negative hotword training data, and (iii) the seed hotsound, performing an optimization of a position of the seed hotsound within the feature space to generate a modified seed hotsound, generating a set of hotsound training data using the modified seed hotsound, training a second neural network to recognize the modified seed hotsound using the generated set of hotsound training data, and using the trained second neural network to recognize the modified hotsound.