The present invention discloses a surgical maneuvering system, in which movement of a moving element is used to control a surgical device such as a surgical tool or an endoscope. The moving element can be a tool or a portion of an operator's body. The relationship between the moving element and the surgical device can be any of: motion of a moving element controlling motion of a surgical device, motion of a moving element controlling an action of a surgical device, a command of a moving element controlling motion of a surgical device, or a command of a moving element controlling an action of the device. Commands are typically arbitrary movements such as shaking a tool. Actions are changes in a surgical device that do not change its overall position, such as closing a grasper.