A method of fabricating tantalum nitride barrier layer in an ultra low threshold voltage semiconductor device is provided. The method includes forming a high-k dielectric layer over a semiconductor substrate. Subsequently, a tantalum nitride barrier layer is formed on the high-k dielectric layer. The tantalum nitride barrier layer has a Ta:N ratio between 1.2 and 3. Next, a plurality of first metal gates is formed on the tantalum nitride barrier layer. The first metal gates are patterned, and then a second metal gate is formed on the tantalum nitride barrier layer.