A residue image down- and/or up-sampling method and apparatus and an image encoding and/or decoding method and apparatus using the residue image down- and/or up-sampling method and apparatus are provided. The residue image downsampling method includes: generating a residue corresponding to the difference between an original image and a predicted image, for each image component of the original image formed with at least two or more image components; and downsampling the residue for each image component at a predetermined ratio. The residue image upsampling method includes: upsampling data downsampled from residue data of an original image; and restoring the original image by adding the predicted image to the upsampled residue of each component. According to the methods and apparatuses, a residue image is obtained by performing spatiotemporal prediction encoding first, and by sampling this residue image, loss of information occurring in the sampling process can be reduced. Since sampling is performed with a residue image obtained through a spatiotemporal prediction process, even when an original image that is not color transformed is directly encoded, sampling can be performed effectively. Also, the methods and apparatuses have an advantage that in addition to colors, sampling of any components can be performed effectively.