A diffractive optical neural network includes one more layers that are resilient to misalignments, fabrication-related errors, detector noise, and/or other sources of error. A diffractive optical neural network model is first trained with a computing device to perform a statistical inference task such as image classification (e.g., object classification). The model is trained using images or training optical signals along with random misalignments of the plurality of layers, fabrication-related errors, input plane or output plane misalignments, and/or detector noise, followed by computing an optical output of the diffractive optical neural network model through optical transmission and/or reflection resulting from the diffractive optical neural network and iteratively adjusting complex-valued transmission and/or reflection coefficients for each layer until optimized transmission/reflection coefficients are obtained. Once the model is optimized, the physical embodiment of the diffractive optical neural network is manufactured.