A system for performing interactions within a physical environment including a robot base that undergoes movement relative to the environment, a robot arm mounted to the robot base, the robot arm including an end effector mounted thereon and a tracking system that measures a robot base position indicative of a position of the robot base relative to the environment. A control system acquires an indication of an end effector destination, and repeatedly determines a robot base position using signals from the tracking system, calculates an end effector path extending to the end effector destination at least in part using the robot base position, generates robot control signals based on the end effector path and applies the robot control signals to the robot arm to cause the end effector to be moved along the end effector path towards the destination.