An electronic device may be provided with a display. The display may have an array of pixels that form an active area and may have an inactive area that runs along an edge of the active area. An opaque layer may be formed on an inner surface of a display cover layer in the inactive area of the display or may be formed on another transparent layer in the electronic device. An ambient light sensor window may be formed from the opening and may be aligned with color ambient light sensor. The ambient light sensor may have an integrated circuit with photodetectors. Ambient light passing through the ambient light sensor window may be diffused by a light diffuser having one or more light-diffusing layers. Diffused ambient light may be collimated using a light collimator having one or more light-collimating layers such as layers with inwardly facing protrusions.