A vehicular driver monitoring system includes an interior rearview mirror assembly disposed in a vehicle and including a mirror reflective element. A camera is disposed at the interior rearview mirror assembly and views a driver sitting at a driver seat of the equipped vehicle. The camera is operable to capture image data. A control includes a vision system-on-a-chip image processor that processes image data captured by the camera. The control, via processing at the vision system-on-a-chip image processor of image data captured by the camera, determines driver movement. The vehicular driver monitoring system generates an output responsive at least in part to the determined driver movement.