Main parts of an endoscope system of the present invention include a medical observation apparatus, a medical image processing apparatus, and a monitor. A CPU of the medical image processing apparatus is constituted by function units including a three-dimensional model estimating unit, a detection target area setting unit, a shape feature value calculating unit, a three-dimensional shape detecting unit, a threshold determining unit, and a polyp determining unit. Such a configuration enables to execute a process appropriately adapted to an observation state of a targeted two-dimensional image and to improve the detection accuracy in the detection of a lesion with locally elevated shape as compared to the past.