An apparatus for decoding an image includes an entropy decoder that performs entropy-decoding to obtain quantized transformation coefficients of at least one transformation unit in a coding unit of the image, an inverse transformer that performs inverse-quantization and inverse-transformation on the quantized transformation coefficients of the at least one transformation unit to obtain residuals, and a restorer that performs inter prediction for at least one prediction unit in the coding unit to generate a predictor and restores the image by using the residuals and the predictor.