Video encoding/decoding apparatus includes video encoder and video decoder. Video encoder generates predicted subblocks by predicting subblocks into which macroblock is divided, generates residual subblocks by subtracting predicted subblocks from subblocks, generates quantized transform coefficients by transforming and quantizing residual subblocks by using transform selected according to size of subblock, generates encoded video data by encoding quantized transform coefficients, and selectively calculates and determines encoding cost when determining encoding mode for at least one of motion vector, transform type, prediction mode, and block type for subblocks. Video decoder reconstructs partition type information, motion information, transform type information, and quantized transform coefficients by decoding encoded data, reconstructs residual subblocks by inverse-quantizing and inverse-transforming quantized transform coefficients by using transform identified by transform type information, generates predicted subblocks by predicting subblocks identified according to partition type information by using motion information, and reconstructs subblocks by adding predicted subblocks and residual subblocks.