A method and apparatus encode and decode video by determining whether to use discrete cosine transform (DCT) and DST for each of the horizontal and vertical transforms. During encoding, an intra-prediction is performed based on an intra-prediction mode determined for an M×N input image block to obtain an M×N intra-prediction residue matrix (E). Based on the intra-prediction mode, each of a horizontal transform and a vertical transform is performed using one of DCT and DST according to the intra-prediction mode. During decoding, the intra-prediction mode is determined from an incoming video bitstream. The M×N transformed coefficient matrix of the error residue is obtained from the video bitstream using an inverse quantizer. Based on the intra prediction mode, one of DCT and DST is performed for each of an inverse vertical transform and an inverse horizontal transform.