A vision system for a vehicle includes a camera disposed at a vehicle equipped with said vision system, the camera having a field of view at least forward of the equipped vehicle. A control includes an image processor for processing image data captured by the camera. The control, via processing by the image processor of image data captured by the camera, determines position of the sun relative to a driver's forward view through the windshield. Responsive to determining, via processing by the image processor of image data captured by the camera, that the position of the sun is likely to hinder the driver's forward view through the windshield, the control controls a sun visor of the equipped vehicle to limit sunlight from reaching the driver's eyes.