A method and apparatus to scalably encode or decode an audio signal or a video signal. A signal distribution is considered after binarization is performed based on a tree structure. Thus, a decoding end can minimize generation of quantization errors and restore an original signal. In addition, only frequency components perceptually important in terms of human hearing are set to an upper layer and encoded without decreasing a bandwidth which is restored from an upper layer. Therefore, sound quality can be increased by decreasing quantization noise while maintaining a relatively large frequency bandwidth, and the degradation of a compression performance is prevented.