The present disclosure generally relates to the control of robotic end-effectors in order to manipulate objects. An exemplary method includes updating a classifier based on sensor data obtained at a first time and applying the updated classifier to second sensor data obtained at a second time, to assess status of a robotic end-effector with respect to one or more objects. The method further includes determining a robotic action based on the status assessed and causing a robotic device including the robotic end-effector to perform the robotic action.