The present invention relates to a sound image localizing apparatus for generating a sound image localized in an arbitrarily set three-dimensional direction. The sound image direction setting unit (111) is operable to set a three-dimensional direction to which the sound image is to be localized in accordance with a listener's instruction; the angle conversion unit (112) is operable to convert the direction into a lateral angle (α) and a rising angle (β), each of which serves as a cue for the listener to perceive the direction. The convolution operation unit 116 is operable to obtain lateral angle control information corresponding to the lateral angle (α) from the lateral angle control information storage unit (113), a listener's front direction head-related impulse response from the front direction head-related impulse response storage unit (115), and rising angle difference information corresponding to the rising angle (β) from the rising angle difference information storage unit (114). The convolution operation unit 116 is then operable to convolute the sound source signal with the lateral angle control information, the front direction head-related impulse response, and the rising angle difference information, and output the operation result.