A focus detection device includes: an image sensor that includes a plurality of focus detection pixel rows, each having a plurality of focus detection pixels and each outputting a pair of focus detection signals; a spatial accumulation unit that calculates spatially accumulated values by adding together pairs of focus detection signals output from a first predetermined number of focus detection pixel rows among the plurality of focus detection pixel rows; and a focus detection unit that detects a focusing condition at an optical system based upon the spatially accumulated values.