Implementations relate to detecting user touch on a controller handle. In some implementations, a non-controlling mode of a control system is activated, and in the non-controlling mode, one or more actuators are controlled to cause a vibration to be provided on a handle of a controller. The vibration is sensed with one or more sensors, and a difference in the vibration is determined to have occurred relative to a reference vibration using the one or more sensors, where the difference satisfies one or more predetermined thresholds. A controlling mode of the system is activated in response to determining the difference in the vibration, and the vibration is modified on the handle in response to detecting the change in the vibration.