A PCM signal processing apparatus for recording or reproducing a PCM signal. The PCM signal reproducing apparatus of the invention is constituted by: a data addition circuit for receiving a first signal composed of n bits per word (n being an integer) and a second signal composed of n' bits per word (n' being an integer) as mode inputs different from each other, and for adding fixed data of (n-n') bits to every word of the second signal to thereby generate a third signal; a first switch device for receiving the first and third signals and for selectively outputting one of the first and third signals; a symbol generation circuit for converting every word of the output of the first switch device into symbol data of m.times.l bits (n>l); a signal processing circuit for performing at least interleave processing, error correction code generating processing, and addition of a synchronizing signal, on the output of the symbol generation circuit, to thereby generate a first data frame; a data deletion circuit for deleting the fixed data from the first data frame of the signal processing circuit to thereby generate a second data frame; and a second switch devices for receiving the first data frame from the signal processing circuit and the second data frame from the data deletion circuit.