A vision system for a vehicle includes a camera disposed at a vehicle and having a field of view exterior of the vehicle. A brightness control, responsive to processing of a frame of image data captured by the camera, interpolates towards an expected brightness for a next frame of captured image data by calculating a set of at least three brightness values using three different control coefficients derived from a previous frame of image data captured by said camera. The brightness control interpolates toward the expected brightness of the next frame of captured image data using the current expected brightness value and two of the three brightness values derived from the three control coefficients derived from the previous frame of captured image data.