The interchangeable lens apparatus (100) provided with an image capturing optical system and being detachably attachable to an image capturing apparatus (200), the image capturing apparatus being configured to detect a defocus amount of the image capturing optical system by using a signal acquired by photoelectric conversion of an optical image formed by the image capturing optical system. The interchangeable lens apparatus includes a focus lens (105) included in the image capturing optical system, and a memory (220) to store information on a focus sensitivity indicating a relation between a unit movement amount of the focus lens and a displacement amount of the optical image. The information on the focus sensitivity is information for enabling acquiring the focus sensitivity different depending on an image height.