In transmission or recording of a digital signal by means of digital amplitude and phase modulation, particularly recording into and reproducing the same from the a magnetic recording medium, a digital amplitude/phase modulated signal to be demodulated is commonly affected by the non-linear amplitude characteristics and the effects of unwanted jitter and envelope variation. As the result, the error rate of decoded data will be increased. For compensation, the Viterbi decoding technique is employed with the use of a weighted distance value calculated by multiplying an Euclidean distance between the signal point of a reproduction signal and the datum point of each code by a positive weighting factor which is determined smaller when the code datum point is located further from the origin point of a coordinate system. Accordingly, the weighted distance of each reproduction signal point from its corresponding datum point can approximately be uniform thus ensuring less error decoding in general.