First and second semiconductor structures, a CESL, and an ILD layer are formed on a substrate. The first semiconductor structure includes first dummy gate, first nitride mask, and first oxide mask. The second semiconductor structure includes second dummy gate, second nitride mask, and second oxide mask. A first planarization is performed to remove a portion of the ILD layer, exposing CESL. A portion of the CESL, a portion of the ILD layer, the first and the second oxide masks are removed. A hard mask layer is formed on the first and the second nitride masks, and in a recess of the ILD layer. A second planarization is performed to remove a portion of the hard mask layer, the first and the second nitride masks, exposing first and second dummy gates. A remaining portion of the hard mask layer covers the ILD layer.