The present disclosure provides systems and methods for controlling autonomous vehicles. In one example implementation, a method includes providing for display to a passenger of an autonomous vehicle, by a computing system having one or more computing devices, a user interface on a display device. The user interface includes at least one interface element associated with directing movement of the autonomous vehicle. The method includes receiving, by the computing system, a user interaction for a time period directed to the interface element. During the time period, the method includes providing, by the computing system, one or more signals indicative of the user interaction to control the autonomous vehicle to autonomously travel along a predetermined path.