A method of detecting a phase difference between a pair of images of an object which are formed on a pair of optical sensor arrays through different optical paths, with presence of flare in one of the images, is disclosed. The method includes the steps of: preparing a sensor data group representing a pattern of each image and consisting of sensor data received from each optical sensor array; correcting the sensor data group so as to eliminate a difference in the average value of the sensor data between the optical sensor arrays; preparing a plurality of combinations each having two subgroups corresponding to the pair of optical sensor arrays, each subgroup consisting of a predetermined number of a series of sensor data, which are selected from the sensor data group corrected, such that a portion of the sensor data group from which the subgroup is selected is sequentially shifted; and selecting one of the combinations having a maximum degree of correlation between the subgroups, and detecting a phase difference from positions on the optical sensor arrays which correspond to the subgroups of the selected combination.