A method of forming an integrated circuit structure includes providing a substrate; forming a first hard mask layer over the substrate; forming a second hard mask layer over the first hard mask layer; patterning the second hard mask layer to form a hard mask; and, after the step of patterning the second hard mask layer, baking the substrate, the first hard mask layer, and the hard mask. After the step of baking, a spacer layer is formed, which includes a first portion on a top of the hard mask, and a second portion and a third portion on opposite sidewalls of the hard mask. The method further includes removing the first portion of the spacer layer; removing the hard mask; and using the second portion and the third portion of the spacer layer as masks to pattern the first hard mask layer.