Computer-implemented methods, carrier media, and systems for selecting polarization settings for an inspection system for inspection of a layer of a wafer are provided. One method includes detecting a population of defects on the layer of the wafer using results of each of two or more scans of the wafer performed with different combinations of polarization settings of the inspection system for illumination and collection of light scattered from the wafer. The method also includes identifying a subpopulation of the defects for each of the different combinations, each of which includes the defects that are common to at least two of the different combinations, and determining a characteristic of a measure of signal-to-noise for each of the subpopulations. The method further includes selecting the polarization settings for the illumination and the collection to be used for the inspection corresponding to the subpopulation having the best value for the characteristic.