To provide an acoustic signal processing apparatus which can reduce the amount of calculation in matrix arithmetic.An acoustic signal processing apparatus (24) converts down-mixed acoustic signals of NI channels to acoustic signals of NO channels, where NO>NI. The acoustic signal processing apparatus includes: a first matrix arithmetic unit (241) for performing arithmetic on a matrix with K rows and NI columns, where NO>K≧NI, for the down-mixed acoustic signals of the NI channels, and outputting K signals obtained after the matrix arithmetic; K decorrelation units (242, 243) for generating signals incoherent, in terms of time characteristics, with the signals obtained after the matrix arithmetic, while maintaining frequency characteristics of the signals obtained after the matrix arithmetic; and a second matrix arithmetic unit (244) for performing arithmetic on a matrix with NO rows and (NI+K) columns for the down-mixed acoustic signals of the NI channels and for the K incoherent signals, and outputting the acoustic signals of the NO channels.