In a focus adjustment device that images a subject that has been subjected to pupil division to generate image data, and carries out a focus adjustment operation based on the image data, a focus adjustment operation is carried out based on a first phase difference amount in the event that it has been determined that the first phase difference amount is larger than a first threshold value, and then by executing focus detection after the focus adjustment operation, in the event that it has been determined that reliability of both the first phase difference amount and the second phase difference amount is high, and that a difference between the two is smaller than a second threshold value, and that it has been determined that the first phase difference amount is smaller than the first threshold value, a focus adjustment operation is carried out based on a second phase difference amount.