Embodiments of the present invention determine the configuration of a workpiece and whether it is actually being handled by a monitored piece of machinery, such as a robot. The problem solved by the invention is especially challenging in real-world factory environments because many objects, most of which are not workpieces, may be in proximity to the machinery. Accordingly, embodiments of the invention utilize semantic understanding to distinguish between workpieces that may become associated with the robot and other objects (and humans) in the workspace that will not, and detect when the robot is carrying a workpiece.