A method of forming a layout of a photomask includes receiving a layout of a mask pattern, obtaining image parameters of a two-dimensional (2D) layout mask from a simulation, obtaining image parameters of a three-dimensional (3D) layout mask from a simulation, and obtaining differences between the image parameters of the 2D and 3D masks. The differences between the image parameters of the 2D and 3D masks can be compensated by convolving a probability function with respect to an open area, represented by a visible kernel function, with a mask function to produce a first function, convolving a probability function with respect to a blocked area, represented by a visible kernel function, with the mask function to produce a second function, and summing the first function and the second function to produce a compensated vector. The layout of the mask pattern can be corrected using the compensated vector.