A signal under measurement is transformed into a complex analytic signal using a Hilbert transformer and an instantaneous phase of this analytic signal is estimated. A linear phase is subtracted from the instantaneous phase to obtain a phase noise waveform. The phase noise waveform is sampled in the proximity of a zero crossing point of a real part of the analytic signal. A differential waveform of the sample phase noise waveform is calculated to obtain a differential phase noise waveform. An RMS jitter is obtained from the phase noise waveform, and a peak-to-peak jitter is obtained from the phase noise waveform.