A phase difference pixel extraction unit reads phase difference pixel data obtained by performing a storing operation in the same storing time for each of the phase difference pixels in each frame of an imaging operation of an image pickup device. A phase difference pixel calculation unit adds the read frame-by-frame phase difference pixel data of the same coordinates in a current frame and a past frame. A determination unit determines whether to perform a focus detection operation based on the added phase difference pixel data. A ranging calculation processing unit applies the added phase difference pixel data when it is determined, and otherwise applies the phase difference pixel data in the current frame.