In a near-eye optical display system comprising a waveguide and diffractive optical elements (DOEs) configured for in-coupling, exit pupil expansion, and out-coupling, a rainbow phenomenon manifested in the display may be removed or reduced using a polarizing filter at the front of the system so that real-world/stray light entering the system has a particular polarization state, for example TM-polarized. The polarizing filter is utilized in conjunction with a downstream out-coupling DOE that includes diffractive grating structures that are configured to enable sensitivity to an opposite polarization state, for example TE-polarized. An imager is configured to produce virtual-world images that also have a TE-polarized state. The polarization-sensitive out-coupling DOE diffracts the TE-polarized imaging beam out of the grating for display while the TM-polarized light from the real world and/or stray light passes through the grating without diffraction and thus cannot contribute to rainbows in the display.