Embodiments provide functionality to prevent collisions between robots and objects. An example embodiment detects a type and a location of an object based on a camera image of the object, where the image has a reference frame. Motion of the object is then predicted based on at least one of: the detected type of the object, the detected location of the object, and a model of object motion. To continue, a motion plan for the robot is generated that avoids having the robot collide with the object based on the predicted motion of the object and a transformation between the reference frame of the image and a reference frame of the robot. The robot can be controlled to move in accordance with the motion plan or a signal can be generated that controls the robot to operate in accordance with the motion plan.