An autonomous driving system includes: a driver information acquisition device that acquires driver information indicating an action and a state of a driver of a vehicle; and a control device that performs autonomous driving control that controls autonomous driving of the vehicle. The autonomous driving control includes: deactivating action detection processing that detects, based on the driver information, a deactivating action of the driver to deactivate the autonomous driving; ready state detection processing that detects, based on the driver information, a ready state indicating that the driver is ready for manual driving; and deactivation processing that deactivates the autonomous driving when the ready state is detected after the deactivating action is detected.