An audio signal processing method and unit for scrambling and descrambling audio signals accompanying video signals. The audio signal processing method comprises steps of dividing digital audio signals into data blocks synchronized to video signals, and then switching the order of adjacent odd and even blocks. The audio signal processing unit comprises a synchronizing signal detector for detecting the synchronizing signal in the video signal; a timing controller for generating a sampling clock signal for A/D conversion, sampling signal for D/A conversion, and system clock from the synchronizing signal; an A/D converter for converting the analog audio signal to digital audio signal using the sampling clock for A/D conversion; a scrambler for dividing the digital audio signal into data blocks using the system clock and switching adjacent odd and even blocks; and a D/A converter for converting the output signal of the scrambler to the analog audio signal using the sampling clock for D/A conversion.