A method of quantization matrix compression in a video encoder is provided that includes preprocessing a quantization matrix by performing at least one selected from down-sampling the quantization matrix and imposing 135 degree symmetry on the quantization matrix, performing zigzag scanning on the pre-processed quantization matrix to generate a one dimensional (1D) sequence, predicting the 1D sequence to generate a residual 1D sequence, and coding the residual 1D sequence using kth order exp-Golomb coding to generate a compressed quantization matrix, wherein k≧0.