A metrology system includes a radiation source that generates light, an optical modulation unit, a reflector, an interferometer, and a detector. The optical modulating unit temporally separates a first polarization mode of the light from a second polarization mode of the light. The reflector directs the light towards a substrate. The interferometer interferes the diffracted light from a pattern on the substrate, or reflected light from the substrate, and produces output light from the interference. The detector receives the output light from the interferometer. The first and second polarization modes of the output light are temporally separated at the detector. Additionally, an optical rotator can be configured to receive the first polarized light and rotate the polarization of the first polarized light.