A driving handover control device includes a memory and a processor coupled to the memory. In a case in which driving is handed over from a first state in which a vehicle travels in accordance with instruction from a remote operator of the vehicle to a second state in which the vehicle travels in accordance with operation by a passenger of the vehicle, the processor implements a transition from the first state to a third state in which the vehicle travels autonomously without instruction from the remote operator or operation by the passenger, and, after the transition from the first state to the third state, implements a transition from the third state to the second state.