A coated article is provided with a low-emissivity (low-E) layer stack for use in a window unit or the like. The layer stack, or coating, may permit the coated article to achieve one or more of a low solar factor (SF) value, a high selectivity (Tvis/SF) value, substantially neutral color at normal and/or off-axis viewing angles, and/or low emissivity. When high selectivity values are achieved, there is provided a high ratio of visible transmission to SF, which is a desirable feature in certain example instances. In certain example embodiments, a sub-oxide layer (e.g., NiCrOx) may be used as a contact layer under an infrared (IR) reflecting layer in order to permit low SF values, high selectivity, and good coloration to be achieved.