Down-sampling of an image may be performed in the DCT domain. A multiple layered network is used to select transform matrices for down-sampling a DCT image of size M×N to a DCT image of size I×J. A spatial domain down-sampling method is selected and applied to the DCT image to produce a down-sampled DCT reference image. A learning with forgetting algorithm is used to apply a decay to the elements of the transform matrix and select a transform matrices which solve an optimization problem. The optimization problem is a function of the visual quality of images obtained using the transform matrices and the computational complexity associated with using the transform matrices. The visual quality is a measure of the difference between the down-sampled DCT image obtained using the transform matrices and the visual quality of the DCT reference image obtained using a spatial domain down-sampling method.