Techniques described herein relate to using reduced-dimensionality embeddings generated from robot sensor data to identify predetermined semantic labels that guide robot interaction with objects. In various implementations, sensor data obtained from one or more sensors of a robot includes data indicative of an object observed in an environment in which the robot operates. The sensor data is processed utilizing a first trained machine learning model to generate a first embedded feature vector that maps the data indicative of the object to an embedding space. Nearest neighbor(s) of the first embedded feature vector is identified in the embedding space. Semantic label(s) are identified based on the nearest neighbor(s). A given grasp option is selected from enumerated grasp options previously associated with the semantic label(s). The robot is operated to interact with the object based on the pose and using the given grasp option.