This method includes optimizing, by a gradient descent method, a first parameter used in back propagation processing and associated with XPM and a second parameter used in the back propagation processing and associated with SPM and XPM, wherein the back propagation processing is processing to estimate a waveform at a time of transmission by alternately calculating linear terms and nonlinear terms in a nonlinear Schrödinger equation after receiving an optical signal whose waveform shape changed in a transmission line and digitizing a waveform of the received optical signal, and correct, for each channel of plural channels in the transmission line at a time of wavelength-division multiplexing transmission, waveform distortion caused by SPM that occurs in the channel and waveform distortion caused by XPM that occurs in relation with channels other than the channel; and executing the back propagation processing by using the optimized first and second parameters.