There is provided an image processing device including a setting section that sets, for respective transform units, a quantization matrix used when inversely quantizing transform coefficient data of an image to be decoded, according to an orthogonal transform method selected when inversely orthogonally transforming the transform coefficient data, an inverse quantization section that uses the quantization matrix set by the setting section to inversely quantize the transform coefficient data, and a transform section that uses the selected orthogonal transform method to inversely orthogonally transform the transform coefficient data inversely quantized by the inverse quantization section.