There is provided with a convolution operation apparatus. A first product-sum calculator includes a plurality of first multipliers, a plurality of first accumulators, a first data supplier which supplies first data to the plurality of first multipliers, and a second data supplier which supplies a plurality of second data to the plurality of first multipliers. A second product-sum calculator includes a plurality of second multipliers, a plurality of second accumulators, a third data supplier which supplies third data to the plurality of second multipliers, and a fourth data supplier which receives and holds an output from each of the plurality of first accumulators and supply the outputs as fourth data to a corresponding one of the plurality of second multipliers.