An image pickup device includes a pixel section in which image pixels and phase difference pixels are arranged in a matrix shape, an image-signal generating section that stores pixel signals outputted from the pixel section in a memory section for one or more rows of unit rows, which are rows of unit pixels configured by a plurality of pixels adjacent to each other, combines, for each of the unit rows, the pixel signals of the unit pixels as combined pixel signals, and extracts phase difference pixel signals from the unit row, and an image-signal readout section that reads out, as signals of one row, the combined pixel signals and the phase difference pixel signals of one unit row generated by the image-signal generating section.