An autonomous driving system (100; 200; 300; 400; 500) performing autonomous driving of a vehicle, includes a vehicle position acquisition device (11) acquiring a position of the vehicle on a map, a vehicle state acquisition device (13) acquiring a vehicle state of the vehicle, an occupant state acquisition device (14) acquiring an occupant state of the vehicle, a destination candidate database (6) storing destination candidate data including a destination candidate and the occupant state correlated with the destination candidate; and an electronic control unit (10; 30; 40; 50; 60). The electronic control unit (10; 30; 40; 50; 60) determines whether or not the vehicle is in a boarding completion state based on the vehicle state and the occupant state, and sets a destination of the vehicle in autonomous driving based on the position of the vehicle on the map, the occupant state, and the destination candidate data in a case where electronic control unit (10; 30; 40; 50; 60) determines that the vehicle is in the boarding completion state.