Provided is an image processing device including a selection section configured to select, from a plurality of transform units with different sizes, a transform unit used for inverse orthogonal transformation of image data to be decoded, a generation section configured to generate, from a first quantization matrix corresponding to a transform unit for a first size, a second quantization matrix corresponding to a transform unit for a second size from a first quantization matrix corresponding to a transform unit for a first size, and an inverse quantization section configured to inversely quantize transform coefficient data for the image data using the second quantization matrix generated by the generation section when the selection section selects the transform unit for the second size.