An autonomous driving device includes an autonomous driving release unit configured to release the autonomous driving control if an override operation is input during the execution of the autonomous driving control which is based on a first travel plan, a travel plan setting unit configured to set a second travel plan different from the first travel plan after the input of the override operation, an autonomous driving restore determination unit configured to determine whether or not to restore the autonomous driving control with the autonomous driving control which is based on the second travel plan after the input of the override operation, and a vehicle control unit configured to maintain the manual driving and prohibit automatic starting of the autonomous driving control until after a predetermined time elapses after the autonomous driving control is released by the autonomous driving release unit and when the autonomous driving restore determination unit determines not to restore the autonomous driving control.