A vehicular rearview mirror control system includes an interior rearview mirror assembly, a driver-side exterior rearview mirror assembly, and a rearward-viewing camera. The system, via processing of a first subset of captured image data, is operable to determine ambient light rearward of the vehicle, and via processing of a second subset of the captured image data, is operable to determine light emanating from a headlight of a following vehicle. Responsive to determining that the light emanating from the following vehicle's headlight is at a driver-side region and glares a driver of the vehicle, the system dims the driver-side exterior rearview mirror assembly to reduce glare to the driver. Responsive to determining that the determined light emanating from the following vehicle's headlight is at a central region and glares the driver of the vehicle, the system dims the interior rearview mirror assembly to reduce glare to the driver.