A vehicle control device (20) has a drive permission unit (29) for selecting between a drive-permitted state and a drive-prohibited state, a drive state detection unit (50) for sequentially detecting which one of the drive-permitted state and the drive-prohibited state is selected, a portable-unit restraining unit (42) for unremovably restraining a portable unit (10) inserted in a slot (21), and a portable-unit restraining control unit (51) which, when the portable unit (10) is inserted in the slot (21) in the drive-permitted state, unremovably fixed the portable unit (10), which is inserted in the slot (21), by the portable-unit restraining unit (42) and, when the portable unit (10) is inserted in the slot (21) in the drive-prohibited state, releases the portable unit (10) from the locking effected by the portable-unit restraining unit (42) to allow removal of the portable unit (10) from the slot (21). According thereto, when the portable unit, such as an electronic key enabling the start of a driving source such as an engine, is inserted in the slot, the vehicle control device allows a safe control on locking of the portable unit to the slot and release of the locking according to situations.