A markerless robot tracking system (10) employing a surgical RCM robot (20) including a primary revolute joint (22) rotatable about a primary rotational axis and a secondary revolute joint (22) rotatable about a secondary rotational axis. A plurality of unique landmark sets are integrated into the robot (20) with each unique landmark set including landmark(s) (30) in a fixed orientation relative to the primary rotational axis and further including additional landmark(s) (30) in a fixed orientation relative to the secondary rotational axis. The system (10) further an optical camera (41, 51) for visualizing a subset of the plurality of unique landmark sets within a camera coordinate system (43, 53), and a robot tracking controller (70) for estimating a robot pose of the surgical RCM robot (20) within the camera coordinate system (43, 53) derived from the visualization by the optical camera (41, 51) of the subset of landmark(s) within the camera coordinate system (43, 53).