The disclosure relates to an audio signal processing apparatus for modifying a stereo image of a stereo signal. The apparatus includes a panning index modifier configured to apply a mapping function to at least all panning indexes of stereo signal time-frequency segments that are within a frequency bandwidth, a first panning gain determiner configured to determine modified panning gains for time-frequency signal segments of the first and second audio signal based on the modified panning indexes, and a re-panner configured to re-pan the stereo signal according to ratios between the modified panning gains and panning gains of the first and second audio signal that correspond to the modified panning gains in time and frequency.