The audio signal processing apparatus includes a processor configured to adjust an output loudness level of an input content. The processor is configured to obtain a loudness information analyzed from the input content, obtain a loudness gain applied to the input content based on the loudness information and a target loudness level, and adjust the output loudness level of the input content based on the loudness gain. In this case, in a first section among an entire section in which the input content is reproduced, the loudness gain is limited based on a first gain allowable range, and in a second section following the first section, the limitation on the loudness gain is released.