A method of forming a semiconductor device includes etching a gate stack to form a trench extending into the gate stack, forming a dielectric layer on a sidewall of the gate stack, with the sidewall exposed to the trench, and etching the dielectric layer to remove a first portion of the dielectric layer at a bottom of the trench. A second portion of the dielectric layer on the sidewall of the gate stack remains after the dielectric layer is etched. After the first portion of the dielectric layer is removed, the second portion of the dielectric layer is removed to reveal the sidewall of the gate stack. The trench is filled with a dielectric region, which contacts the sidewall of the gate stack.