A method includes forming a dummy gate stack, etching the dummy gate stack to form an opening, depositing a first dielectric layer extending into the opening, and depositing a second dielectric layer on the first dielectric layer and extending into the opening. A planarization process is then performed to form a gate isolation region including the first dielectric layer and the second dielectric layer. The dummy gate stack is then removed to form trenches on opposing sides of the gate isolation region. The method further includes performing a first etching process to remove sidewall portions of the first dielectric layer, performing a second etching process to thin the second dielectric layer, and forming replacement gates in the trenches.