A phase difference detection apparatus for detecting a phase difference between images formed on a pair of optical sensor arrays in which the calculation of a compensation value is simplified. When a compensation effectiveness judgment unit judges that compensation of the pair of image data rows would be effective, a left and right difference compensation unit calculates a compensation amount based on a difference in maximum values and minimum values in a pair of image data rows corresponding to the images produced by the optical sensor arrays and compensates the image data rows by the calculated compensation amount. Based upon the compensated values, a correlation calculation unit carries out a correlation calculation, and a maximum correlation detection unit detects a maximum correlation level. Based upon the maximum correlation level, a interpolation calculation unit carries out an interpolation calculation, and a phase difference is detected by a phase difference detection unit.