An ambient light sensor system may be mounted in alignment with a window in a display cover layer associated with a display in an electronic device. The ambient light sensor system may have a light diffuser layer and an infrared-light-blocking filter. The light diffuser layer may have a polymer layer with embedded light-scattering desiccant particles. An ambient light sensor in the ambient light sensor system may receive ambient light through the light diffuser layer and the infrared-light-blocking filter. The infrared-light-blocking filter may have a polymer substrate and a thin-film interference filter formed from a stack of inorganic thin-film layers on the polymer substrate. Light-scattering desiccant particles may be incorporated into the polymer substrate of the infrared-light-blocking filter. Desiccant may also be incorporated into ambient light sensor support structures.