The invention provides an optimization method of transmission performance of a long-distance coherent optical communication system for solving the problem of difficulty in quickly carrying out global optimization on the transmission performance of the coherent optical communication system with a long distance, a large cross-section number, unequal cross-section indexes and no dispersion compensation. The optimization method comprises the following steps that an optimal performance parameter of the whole optical communication link is computed based on link attention and amplifier noise characteristics; ranges and initial values of one group of weighting coefficients (m, t) are arranged; two factors including an optimal difference value of amplifier gain and an amplifier cross-section position are considered to arrange a module Mea (m, t, n), the corresponding Kminth amplifier cross-section position Mk is selected when the Mea is larger, the amplifier gain is Aopt, Mt, wherein k is equal to 1-Kmin, and the other amplifier gain is the initial value thereof. According to the optimization method, promotion of a light path performance index factor Q is considered, instantaneous characteristics, such as shake and time delay, which are caused by overlarge regulated quantity or overlarge regulation number, are avoided, and the performance of the optical transmission system is quickly optimized under the conditions of the extra-long distance and the large cross-section number.