Provided is an apparatus and method for encoding and decoding using alternative transform units according to the correlation of residual signals. The video encoding apparatus includes a first transforming unit for performing discrete cosine transform (DCT), first quantization, first inverse quantization, and inverse DCT on a block basis onto residual coefficients generated after intra frame prediction or inter frame prediction; a second transforming unit for performing discrete sine transform (DST), second quantization, second inverse quantization, and inverse DST on a block basis onto the residual coefficients; a selecting unit for selecting one having a high compression rate between the first and second transforming units for each block through performing rate-distortion optimization; and a flag marking unit for recording information about the selected transforming unit at a flag bit provided on a macroblock basis.