An audio signal processing apparatus includes a cutout unit, a correlation calculating unit, a spectrum calculating unit, a coefficient calculating unit, and an assigning unit. The cutout unit cuts out audio signals of plural channels by a time frame. The correlation calculating unit calculates a correlation value between respective signals of the plural channels included in a predetermined time frame cut out by the cutout unit. The spectrum calculating unit calculates spectrum information indicative of spectral characteristics concerning a signal of a given channel cut out by the cutout unit. The coefficient calculating unit calculates a coefficient to be multiplied by the signal of the given channel, based on the correlation value and the spectrum information. The assigning unit multiplies the coefficient by the signal of the given channel and assigns the multiplied signal to channels other than the given channel.