An image processing device including an acquiring section configured to acquire quantization matrix parameters from an encoded stream in which the quantization matrix parameters defining a quantization matrix are set within a parameter set which is different from a sequence parameter set and a picture parameter set, a setting section configured to set, based on the quantization matrix parameters acquired by the acquiring section, a quantization matrix which is used when inversely quantizing data decoded from the encoded stream, and an inverse quantization section configured to inversely quantize the data decoded from the encoded stream using the quantization matrix set by the setting section.