Methods, systems, and apparatus, including computer programs encoded on computer storage media, for controlling a robot using symbolic states. One of the methods includes receiving a definition of a task having multiple task states, wherein each task state is associated with a different respective control policy; executing the task using an initial control policy associated with an initial task state; during execution of the task, continually generating, from sensor data, a prediction of the task state of the task; and upon determining that a transition to a different task state has occurred, transitioning the robot to a different control policy associated with the different task state.