A modulation method of an optical modem and a signal transmission apparatus performing the method are disclosed. The modulation method of the optical modem includes an optical interface providing a signal to a light source, a photo detector receiving reflected light by an optical link when output light from the light source based on the provided signal is reflected by the optical link, the photo detector measuring and determining characteristics of the optical link using the reflected light, and the optical modem determining a power level and a modulation method for each subcarrier based on the characteristics of the optical link.