The data processing device includes the inference processor and learning processor. The inference processor includes a input data determination circuit for determining whether or not each of the binarized input data is a predetermined value, a memory for storing a plurality of coefficient and a coefficient address information including information about a coefficient address in which a plurality of coefficient are stored, an inference controller for reading coefficient address from the storage unit based on a determination result of the input data determination circuit and reading coefficient from the storage unit based on coefficient address, a arithmetic circuit for performing an operation using the binarized input data and coefficient acquired by the inference controller to generate the arithmetic operation result as a output data.