A system includes manipulators and a controller. The controller is configured to detect mounting of an imaging device to a first manipulator of the manipulators, determine a first reference frame for the imaging device based on the mounting of the imaging device to the first manipulator, control a tool relative to the first reference frame by controlling a relative position and orientation of a tip of the tool relative to the imaging device in the first reference frame by correlating movement of a master input control to movement of the tool in the first reference frame, detect mounting of the imaging device to a second manipulator of the manipulators, the second manipulator being different from the first manipulator, determine a second reference frame for the imaging device based on the mounting of the imaging device to the second manipulator, and control the tool relative to the second reference frame.