Systems and methods are provided for controlling a vehicle. A safe envelope driving pattern is determined to control the vehicle in an autonomous mode. User identification data and sensor data are received from one or more sensors associated with the vehicle. A driver-specific driving pattern is determined based on the received sensor data and the user identification data. Operation of the vehicle is controlled in the autonomous mode based on the identification of the user in the driver's seat, the safe envelope driving pattern, and the user-specific driving pattern.