A method includes depositing a dummy gate dielectric layer over a semiconductor region, depositing a dummy gate electrode layer, and performing a first etching process. An upper portion of the dummy gate electrode layer is etched to form an upper portion of a dummy gate electrode. The method further includes forming a protection layer on sidewalls of the upper portion of the dummy gate electrode, and performing a second etching process. A lower portion of the dummy gate electrode layer is etched to form a lower portion of the dummy gate electrode. A third etching process is then performed to etch the lower portion of the dummy gate electrode using the protection layer as an etching mask. The dummy gate electrode is tapered by the third etching process. The protection layer is removed, and the dummy gate electrode is replaced with a replacement gate electrode.