Systems and methods are presented to enhance and isolate residual signals indicative of the speckle field based on measurements taken by optically based metrology systems. Structural irregularities such as roughness and topographical errors give rise to light scattered outside of the specularly reflected component of the diffracted light. The scattered light interferes constructively or destructively with the specular component in a high numerical aperture illumination and detection system to form a speckle field. Various methods of determining residual signals indicative of the speckle field are presented. Furthermore, various methods of determining structural irregularities based on analysis of the residual signals are presented. In various embodiments, illumination with a high degree of spatial coherence is provided over any of a wide range of angles of incidence, multiple polarization channels, and multiple wavelength channels. In addition, diffracted light is collected over a wide range of angles of detection.