An apparatus and method for decoding audio data. The apparatus for decoding the audio data may perform block data unpacking by preferring a channel order to a block order from a bitstream, and perform dithering through preferring a block order to a channel order. Complexity in decoding may be reduced through integrating bitstream searching and the bock data unpacking, and a dithering error may be prevented through processing the block data unpacking and the dithering separately.