A method of decoding an image includes performing entropy-decoding to obtain quantized transformation coefficients of at least one transformation unit in a coding unit of the image, determining a prediction mode of at least one prediction unit in the coding unit from information indicating a prediction mode for the at least one prediction unit, when the prediction mode is determined to be an inter prediction mode, not in an intra prediction mode, determining a size of the at least one transformation unit in the coding unit regardless of a size of the at least one prediction unit in the coding unit, performing inverse-quantization and inverse-transformation on the quantized transformation coefficients of the at least one transformation unit to obtain residuals, and performing inter prediction for at least one prediction unit in the coding unit to generate a predictor and restoring the image by using the residuals and the predictor.