A focus detection device comprises: an image shift detection unit that detects a relative shift amount of a pair of images formed by a pair of light fluxes having passed through an optical system; and a conversion unit that converts the shift amount to a defocus amount based upon dimensional information of an exit pupil corresponding to an aperture restricting light flux in the optical system and distance information of the exit pupil indicating distance to the exit pupil from a predetermined imaging plane of the optical system.