A focus detection apparatus for performing phase difference detection type focus detection includes sensor units each configured to generate a signal used for detecting a phase difference, a setting unit configured to set one or more sensor units as determination targets, a determination unit configured to sequentially select a sensor unit from the determination targets, and repeatedly determine whether a signal generated by the selected sensor unit satisfies a termination condition of having a peak value greater than a threshold, and a detection unit configured to perform focus detection using the signal that satisfies the termination condition. The apparatus operates in a first mode and then shift to a second mode. The setting unit includes, in the determination targets in the second mode, a sensor unit that is not included in the determination targets in the first mode.