An image coding device is provided with a determination unit which determines whether to apply an orthogonal transform to a transform block obtained by dividing a prediction difference signal indicating a difference between an input image and a predicted image or perform a transform skip by which the orthogonal transform is not applied, and an orthogonal transform unit which performs processing selected on the basis of the determination, the image coding device comprising a quantization unit which, when the transform skip is selected on the basis of the determination, quantizes the transform block using a first quantization matrix in which the quantization roughnesses of all elements previously shared with a decoding side are equal, and when the orthogonal transform is applied to the transform block on the basis of the determination, quantizes the transform block using the first quantization matrix or a second quantization matrix that is transmitted to the decoding side.