A color image and video encoding and/or decoding apparatus, method, and medium using inter-color-component prediction in accordance with an encoding mode are provided. The encoding apparatus for encoding an original image including at least two color image components, includes: a residue generation unit, if the encoding mode is inter mode, generating a temporal residue by obtaining the difference of a current image and a prediction image obtained by estimating a motion in units of blocks between the current image and a previous image of each color component; a residue color prediction unit encoding the temporal residue of the residue generation unit, by performing residue inter-color-component prediction; an intra color prediction unit, if the encoding mode is intra mode, encoding the original image by performing inter-color-component prediction of the original image; and an entropy encoding unit generating a bitstream by entropy encoding the result value of the residue color prediction unit or the intra color prediction unit. According to the apparatus, method, and medium, a color image or video data can be directly compressed effectively without a conventional color transform process. Furthermore, by using the correlation of image components, redundant information between color components varying with respect to the encoding mode is removed so that the encoding efficiency can be enhanced.