An object or person in a medical environment may be identified based on images of the medical environment. The identification may include determining an identifier associated with the object or the person, a position of the object or the person in the medical environment, and a three-dimensional (3D) shape/pose of the object or the person. Representation information that indicates at least the determined identifier, position in the medical environment, and 3D shape/pose of the object or the person may be generated and then used (e.g., by a visualization device) together with one or more predetermined 3D models to determine a 3D model for the object or the person identified in the medical environment and generate a visual depiction of at least the object or the person in the medical environment based on the determined 3D model and the position of the object or the person in the medical environment.