An accessory apparatus is removably attached between an image capturing apparatus and an interchangeable lens. The accessory apparatus performs first communication with the image capturing apparatus and second communication with the interchangeable lens, and accepts a predetermined operation related to a manual focus operation. The accessory apparatus sets a degree to which a drive amount of a focus lens in the interchangeable lens is made effective relative to an operation amount of the first operation unit, and transmits the drive amount or a drive speed of the focus lens in the interchangeable lens to the interchangeable lens through the second communication in accordance with the predetermined operation and the degree set.