An optical transmission system according to an embodiment includes: an optical transmission unit which modulates an optical signal into which a known signal is inserted and transmits the optical signal; and an optical reception unit which receives the optical signal from the optical transmission unit, wherein the optical reception unit includes: an optical receiver which performs coherent detection of a reception signal of the optical signal received from the optical transmission unit; a receiver transfer function estimation section which estimates a nonlinear response transfer function of the optical receiver based on the known signal included in the reception signal after the detection by the optical receiver; and a receiver compensation section which compensates nonlinear distortion of the reception signal after the detection based on the nonlinear response transfer function which the receiver transfer function estimation section estimates.