An electronic device such as a watch may include a display and a light sensor located behind the display. The light sensor may be used to measure the color of external objects. During color sampling operations, the display may emit light towards the external object in front of the display while the light sensor gathers color measurements. The display may emit light of different colors and the light sensor may detect an amount of reflected light for each color, which in turn may be used to determine the color of the external object. The control circuitry may use a watch-band-specific algorithm to determine the color of watch bands and may use a clothing-specific algorithm to determine the color of clothing. The control circuitry may display the color on the display so that the face of the watch matches the user's clothing or matches the user's watch band.