Occupancy in a vehicle is determined by maintaining a set of occupant regions detected in a video. An occupant region can be a bounding box around a face, and/or an occupied seat that does not overlap any bounding box. A count, specific to an occupant region, is set to zero when an overlap between the occupant region in a current frame in the video and a previous frame in the video, satisfies a first predetermined condition. When the overlap does not satisfy the first predetermined condition, the count which is specific to the occupant region is incremented and checked against a threshold in a second predetermined condition. When the count exceeds the threshold, that occupant region is removed from the set of occupant regions. The just-described operations are repeated, with additional occupant regions. A count of occupant regions currently in the set may be displayed or transmitted to a server.