A method of operating a surgical control system comprises generating an image of a surgical environment from a viewpoint of an imaging tool. The image includes a field of view. The method further comprises displaying the image on a display system configured to be mounted to a head of a user. The method further comprises detecting a movement of the head of the user and determining if the movement of the user's head is within a boundary corresponding to the field of view. The method further comprises, if the movement of the user's head is determined to be within the boundary: generating a changed image of the surgical environment from a changed viewpoint of the imaging tool; and displaying the changed image on the display system. The changed image includes a changed field of view corresponding to the detected movement of the user's head.