A helical gear planet transmission system with tooth surface modification includes two engaged transmission gears of an inner helical gear pair and an outer helical gear pair, wherein an outer helical gear in the planet transmission system employs non-involute profile after the tooth surface modification, while an inner helical gear still employs the standard involute profile. The helical gear planet transmission system with the tooth surface modification includes the inner and the outer helical gear pairs which are transmitted through a parallel shaft. As the outer helical gear pair is the gear modified by a plunge grinding method, a " local conjugate " phenomenon containing a certain transmission error exits during the engagement of the gear pair, after application of the load, the vibration and noise of a small planetary gear transmission can be effectively reduced, thereby lifting the integral performance of high speed heavy duty planetary gear speed reducer (increaser) in our country.