An arithmetic processing apparatus performs arithmetic by a neural network in which multiple processing layers are hierarchically connected. The arithmetic processing apparatus corresponding to one of the multiple processing layers includes a convolution arithmetic portion and a pooling processing portion. The convolution arithmetic portion receives an input data from another of the plurality of processing layers, performs convolution arithmetic to the input data, and in each arithmetic cycle, outputs a part of all convolution arithmetic result data required for single pooling processing. The pooling processing portion performs the single pooling processing to the all convolution arithmetic result data before executing activation processing.