The structure of a semiconductor device with different gate structures configured to provide ultra-low threshold voltages and a method of fabricating the semiconductor device are disclosed. The method includes forming first and second nanostructured channel regions in first and second nanostructured layers, respectively, and forming first and second gate-all-around (GAA) structures surrounding the first and second nanostructured channel regions, respectively. The forming the first and second GAA structures includes selectively forming an Al-based n-type work function metal layer and a Si-based capping layer on the first nanostructured channel regions, depositing a bi-layer of Al-free p-type work function metal layers on the first and second nanostructured channel regions, depositing a fluorine blocking layer on the bi-layer of Al-free p-type work function layers, and depositing a gate metal fill layer on the fluorine blocking layer.