An image pickup device includes a group of photoelectric conversion cells that output distance-measurement signals for phase difference detection. Each photoelectric conversion cell includes a photodetector and a pupil restricting unit. The photodetector generates the distance-measurement signal. The pupil restricting unit restricts a size of a pupil area, from which arrival light has exited, to a predetermined size in an exit pupil of a taking optical system, object light exiting from the exit pupil of the taking optical system, the arrival light arriving at the photodetector. The predetermined size is less than half a size of an entire area of the exit pupil.