A method includes: decoding a bitstream and thereby outputting a transform coefficients for each color component of the block, a first flag indicating for each color component whether the block includes a non-zero transform coefficient, and a second flag indicating whether the block has been encoded using a color space transform that transforming a color space of a prediction residual from a color space of the original image to another color space; performing a color space inverse transform for the prediction residual restored from the transform coefficients, when the second flag indicates that the block has been encoded using the color space transform; and determining whether to perform chroma residual scaling for the prediction residual of the chrominance component, based on the first flag of a chrominance component and the second flag, the chroma residual scaling that performs scaling based on a luminance component corresponding to the chrominance component.