An optical transmission system, where in an optical transmitter a detection bit having a specific pattern set according to the number of bits to be transmitted within one symbol time, is imparted with respect to a transmission signal in which transmission information has been encoded according to a preset format, and an optical signal generated by modulating light according to the transmission signal is transmitted to a transmission line. In an optical receiver, logic inversion or bit swap of received data is detected and compensated by using the detection bit included in the received signal, a decoding process of the compensated received signal is executed. As a result, when an optical signal capable of transmitting multi-bit information within one symbol time is transferred, it is possible to realize excellent transmission characteristics, by reliably compensating an error in received data caused by the modulation format or the multiplex system of the optical signal.