A control device associated with an autonomous vehicle detects that a person is altering traffic on a road using a hand signal. The control device determines an interpretation of the hand signal. The control device determines a proposed trajectory according to the interpretation of the hand signal. The control device transmits the proposed trajectory to an oversight server. The oversight server determines whether the hand signal is in use to alter the traffic. The oversight server also determines whether the proposed trajectory causes the autonomous vehicle to go out of a pre-mapped area where the autonomous vehicle is able to autonomously travel. In response to determining that the hand signal is in use and that the proposed trajectory does not cause the autonomous vehicle to go out of the pre-mapped area, the oversight server transmits an instruction that indicates to perform the proposed trajectory to the control device.