An adjustment device for a hearing aid includes: an audio extraction section for, through an acoustic process of a collected audio signal, utilizing information of a phoneme or syllable contained in the audio signal to output time information, the time information identifying a point of uttering the phoneme or syllable; an electroencephalogram measurement section for measuring an electroencephalogram signal of a user; a determination section for determining a difficulty of hearing of the phoneme or syllable by relying on an event-related potential in the electroencephalogram signal based on the identified point of uttering the phoneme or syllable as a starting point; a phoneme identification section for, when a plurality of phonemes or syllables are determined as difficult to hear, identifying a chronologically preceding phoneme or syllable among them to be difficult to hear; and a gain adjustment section for, with respect to the phoneme or syllable identified by the phoneme identification section, determining a gain adjustment method based on a type of the phoneme or syllable, and adjusting a gain for the phoneme or syllable in accordance with the determined gain adjustment method.