The invention provides a signal compensation method and device in an optical fiber communication transmission system. The method comprises the following steps: S100, dividing an initial light wave into X-path polarized light and Y-path polarized light; S101, acquiring a probabilistic QAM (Quadrature Amplitude Modulation) signal, and modulating the X-path polarized light through the probabilistic QAM signal to generate a probabilistic QAM modulation signal; and S102, synthesizing the probabilistic QAM modulation signal and the Y-path polarized light into a polarized multiplexing signal, and transmitting the polarized multiplexing signal to a signal receiving end, so that the signal receiving end compensates for a damaged probabilistic QAM modulation signal according to the polarized multiplexing signal, wherein the damaged probabilistic QAM modulation signal is a signal generated by the probability QAM modulation signal after being subjected to polarization rotation RSOP (Resultant Setof Policy) damage in an optical fiber. Through adoption of the method and the device provided by the invention, the demand of the system for the OSNR (Optical Signal to Noise Ratio) is lowered, the damaged signal which is subjected to the RSOP damage is tracked and compensated. Compared with a conventional RSOP equalization algorithm, the signal compensation method and device have higher flexibility, and are suitable for high-speed optical fiber communication network systems interconnected by a future data center.