The invention relates to a back-to-back direct current transmission system based on a virtual synchronous machine and a control method thereof. The system comprises two inverters directly connected tothe direct current side. The difference between the collected actual direct current voltage and the set direct current voltage command is regulated by a regulator to generate a first power difference. Inverting the second mechanical power for active power regulation in the virtual synchronous machine control mode of the second converter, and summed with the first power difference, a first mechanical power which performs active power regulation in a virtual synchronous machine control mode as a first converter, First, the second mechanical power is used for generating the electric angle of thealternating current of the first and second converters. The back-to-back HVDC transmission system based on the virtual synchronous machine technology of the invention can effectively improve the frequency modulation capability of the system and ensure the safe and stable operation of the electric power system under the condition that the installed capacity demand of the generator set is reduced and the equivalent rotational inertia of the electric network is reduced.