The present invention relates to an optical transmission system to which a spatial multiplexing optical fiber is applied as a transmission line. The optical transmission system performs optical transmission using the spatial multiplexing optical fiber as an optical transmission line for transmission of signal light. Namely, the optical transmission is carried out in accordance with a modulation format in which a Q-factor of the signal light propagating in certain propagation mode light propagating in the spatial multiplexing optical fiber and a ratio of a signal light power to a square of the shortest distance between modulation symbols in the modulation format satisfy a predetermined relational expression.