Prism-coupling systems and methods for characterizing large depth-of-layer waveguides are disclosed. The systems and methods utilize a coupling prism having a coupling angle α having a maximum coupling angle αmax at which total internal reflection occurs. The prism angle α is in the range 0.81αmax≦α≦0.99αmax. This configuration causes the more spaced-apart lower-order mode lines to move closer together and the more tightly spaced higher-order mode lines to separate. The adjusted mode-line spacing allows for proper sampling at the detector of the otherwise tightly spaced mode lines. The mode-line spacings of the detected mode spectra are then corrected via post-processing. The corrected mode spectra are then processed to obtain at least one characteristic of the waveguide.