A semiconductor device with a self-aligned contact and a method of manufacturing the same, wherein the method comprises the step of forming a 1st dielectric layer on gate structures, form a self-aligned contact trench between two gate structures, forming an 2nd dielectric layer on the 1st dielectric layer and in the self-aligned contact trench; patterning the 2nd dielectric layer into a 1st portion on the 1st dielectric layer and a 2nd portion filling in the self-aligned contact trench, using the 2nd dielectric layer as a mask to etch the 1st dielectric layer, and forming a metal layer and a self-aligned contact simultaneously in the 1st dielectric layer and in the self-aligned contact trench.