A system and method for in-service transmitter phase noise measurements determines phase nonlinearity by way of comparing unfiltered signal samples and regenerated reference signal samples to calculate phase nonlinearity error. Linear distortion is removed from the received signal samples in order to truly characterize nonlinear behavior of the transmitter. The unfiltered signal samples are generated without applying the receiver shaping filtering. Reference signal samples are regenerated from estimated transmitted symbols derived from the unfiltered signal samples. The transmitted symbols are estimated using a segmented slicer which dynamically estimates constellation decision levels from the unfiltered signal samples. A weighted, least-square based polynomial regression is performed on phase error samples of the unfiltered signal samples to estimate a phase nonlinear error function while suppressing the impact of other non-systematic distortions. The nonlinear phase error is removed from the phase error to produce phase noise containing carrier phase jitter and additive noise. A threshold is determined for replacing phase error values outside the threshold with random values having a distribution equivalent to the carrier phase jitter. The carrier phase jitter may be displayed as a power spectrum of the phase noise.