Optical signal to noise ratio within a band of interest (in-band OSNR) is calculated by using a reference signal for noise estimation. In-band noise at a node along the optical communication path is estimated by subtracting the reference signal contribution from the received in-band signal energy. Contribution from the reference signal is calculated using an effective transfer function of the optical communication path using either a direct method in which measurements are made a priori on an equivalent optical system or an indirect method in which the effective transfer function is calculated using computerized simulations. The selection of which method to use may be based on the desired resolution bandwidth for the estimation of transfer function.