An endoscope system includes: a light source device for sequentially emitting illumination light of three colors; an image sensor having pixels to perform photoelectric conversion on light from an object irradiated with the illumination light and output image signals; an illumination controller for causing the light source device to emit the illumination light of two colors during a one-line exposure period of the image sensor; and a calculation unit for calculating each color component signal for each horizontal line, based on a relational expression using an output value of a horizontal line, among output values of the image signals in a latest frame, a frame located one frame before the latest frame, and a frame located two frames before the latest frame, using each illumination period per one-line exposure period of each color in each frame, and using each exposure period of each color with respect to the horizontal line.