The technology disclosed herein is directed to a medical manipulator system having a medical device. The medical device such as endoscope or other surgical instrument includes an elongated insertion portion. A holder assembly includes an articulated arm carrying the medical device that is attached thereto and is actuated following movement of the medical device and holds the medical device in a stationary state at a desired position. A position/attitude detection unit detects positions and attitudes of the elongated insertion portion in plurality states into which the medical device is caused to swing about a constraint point. A constraint point estimation unit estimates a position of the constraint point based on the positions and attitudes of the insertion portion as detected by the position/attitude detection unit.