One embodiment of the present invention provides a system that reduces computational complexity in simulating an image resulting from an original mask and an optical transmission system. During operation, the system obtains a set transmission cross coefficient (TCC) kernel functions based on the optical transmission system, and obtains a set of transmission functions for a representative pattern which contains features representative of the original mask. The system constructs a new set of kernel functions based on the TCC kernel functions and the transmission functions for the representative pattern, wherein responses to the new kernel functions in a resulting image corresponding to the representative pattern are substantially uncorrelated with one another. The system further produces an intensity distribution of a resulting image corresponding to the original mask based on the new kernel functions, thereby facilitating prediction of a layout that can be produced from the original mask.