A passenger counting system that can count correctly the number of persons present in a vehicle, including persons sitting on the back seat, is provided. The passenger counting system includes an image capturing device 11 which is arranged outside the vehicle and which captures an image of the vehicle from the lateral direction of the vehicle, an image acquisition unit 31 that controls the image capturing device 11 and acquires an image in which the inside of the vehicle is captured as an input image, a profile detection unit 32 that detects a profile of a person from a partial image obtained from the input image based on an image feature amount calculated by using the partial image and outputs a detection result, and a passenger number determination unit 33 that determines the number of persons present in the vehicle based on the detection result.