An image decoding device and an encoding device include an arithmetic unit for performing arithmetic processing, an arithmetic data storage unit for storing an arithmetic result by the arithmetic unit, an input selection unit for selecting whether to read pixel data that is to be inputted to the arithmetic unit from compressed image data or from pixel data stored in the arithmetic data storage unit, and inputting the read pixel data to the arithmetic unit, and an arithmetic control unit for controlling, based on a transform mode used and the number of arithmetic operations in the arithmetic unit, a destination from which the pixel data that is to be inputted to the arithmetic unit by the input selection unit is read as well as a combination of pieces of pixel data targeted for the arithmetic processing by the arithmetic unit and multiplier coefficients for the arithmetic processing.