An electronic device may be provided with a display mounted in a housing. The display may have an array of pixels that form an active area and may have an inactive area that runs along an edge of the active area. An opaque layer may be formed on an inner surface of a display cover layer in the inactive area of the display or may be formed on another transparent layer in the electronic device. An ambient light sensor window may be formed from the opening and may be aligned with color ambient light sensor. The ambient light sensor may have an integrated circuit with an array of photodetectors and may have a color filter layer forming a corresponding array of thin-film interference color filters with different respective pass bands. The color filter layer may have a shared dielectric stack and multiple color-filter-specific dielectric stacks on the shared dielectric stack.