The present invention discloses a transmitter and receiver for optical communications system, which provide compensation of the optical link nonlinearity. M-PSK modulating is used for data embedding in an optical signal in each WDM channel using orthogonal frequency division multiplexing (OFDM) technique. At the receiver side electrical output signals from a coherent optical receiver are processed digitally with the link nonlinearity compensation. It is followed by the signal conversion into frequency domain and information recovery from each subcarrier of the OFDM signal. At the transmitter side an OFDM encoder provides a correction of I and Q components of a M-PSK modulator driving signal to compensate the link nonlinearity prior to sending the optical signal to the receiver.