A medical system includes: an endoscope and at least one treatment tool; a treatment-tool coordinate calculating unit that extracts the treatment tool by processing two or more images acquired at different times by the endoscope, that determines directions of longitudinal axes of the extracted treatment tool, and that calculates a coordinate of an intersection of the determined two or more longitudinal axes; and a judgment unit that judges whether the treatment tool serves as a follow target, on the basis of the coordinate of the intersection calculated by the treatment-tool coordinate calculating unit.