An acoustic signal processing device 1 includes: a focal point position determination unit 12 that obtains a plurality of sets of initial focal point coordinates, coordinates of the virtual sound source, and a direction of directivity thereof, and for a pair of sets of initial focal point coordinates with different polarities among the plurality of sets of initial focal point coordinates, multiplies the sets of initial focal point coordinates by a rotation matrix based on the coordinates of the virtual sound source to thereby determine sets of focal point coordinates, the rotation matrix being specified from the direction of the directivity; a circular harmonic coefficient conversion unit 13 that calculates weights to be applied to multipoles including the sets of focal point coordinates from a circular harmonic coefficient; a filter coefficient computation unit 14 that, for each of the speakers in the speaker array, computes a weighted driving function to be applied to the speaker from the sets of focal point coordinates, polarities of the sets of focal point coordinates, and the weights to be applied to the multipoles; and a convolutional operation unit 15 that, for each of the speakers in the speaker array, convolves the weighted driving function for the speaker into the input acoustic signal to output the output acoustic signal for the speaker.