A focus control device in an endoscope system, implements an autofocus operation in an appropriate state by setting a focus mode of an imaging optical system. The focus control device includes a focus control section that controls a focus of an imaging optical system that includes at least a zoom lens that adjusts an optical magnification, and sets a focus mode of the imaging optical system, and an image acquisition section that acquires an image through the imaging optical system, the focus mode including a fixed focus mode and an autofocus (AF) mode, and the focus control section switching the focus mode between the fixed focus mode and the AF mode corresponding to whether the zoom lens is positioned on a wide-angle side or a telescopic side relative to a reference point that is situated between a wide-angle end and a telescopic end.