A direction judgment unit calculates a first evaluation value based on an image signal of a first focus detection region among focus detection regions to judge a drive direction of a focus lens to be in focus and calculates a second evaluation value based on an image signal of at least one second focus detection region smaller than the first region, the focus detection regions being at least partly the same and being different in size. A control unit determines whether a change of the second evaluation value is a monotonous change when the focus lens is slightly driven a first number of times in a first direction judged on the basis of the first evaluation value and the focus lens is slightly driven a second number of times smaller than the first number of times in a second direction opposite to the first direction, and the control unit performs focus adjustment on the basis of the second evaluation value when the change of the second evaluation value is not a monotonous change.