A focus adjustment device includes an acquisition unit configured to acquire focus evaluated values of a plurality of color signals obtained from an image signal generated by an image sensor that light transmitted through a focus lens has entered, a calculation unit configured to calculate at least one value indicating a difference in magnitude between each pair of the focus evaluated values of the plurality of color signals, and a determination unit configured to determine, in a case in which a value calculated by the calculation unit is within a predetermined range, a drive direction of the focus lens based on the value calculated by the calculation unit, and determine, in a case in which the value calculated by the calculation unit is outside the predetermined range, a drive direction of the focus lens based on a luminance signal obtained from the image signal.