An electronic device may be provided with a display mounted in a housing. The display may have an array of pixels that form an active area and may have an inactive area that runs along an edge of the active area. A color ambient light sensor may gather ambient light measurements through a window in an opaque masking layer in the inactive area. The color ambient light sensor may have photodiodes with different spectral sensitivities and may have a dark current photodiode that is insensitive to light. A set of analog-to-digital converters may be used to digitize photodiode signals from the photodiodes. A switch array may be used to distribute signals from the photodiodes to each of the analog-to-digital converters. This allows the output of each photodiode to be averaged over multiple analog-to-digital converters to remove any impact of variations in performance between converters.