Visually optimized quantization is described herein. Specifically, the visually optimized quantization is for arbitrary block-based transforms such as Mode-Dependent Directional Transform (MDDT). The compression method involves the process of taking a discrete cosine transform of an MDDT basis function, obtaining the frequency weights, and computing the contrast sensitivity function for each of the frequency components. The overall effect of the distortion is calculated by error pooling and the quantization matrix is the inverse proportional of the overall effect.