A semiconductor device a method of forming the same are provided. The semiconductor device includes a substrate, a first isolation structure and a second isolation structure over the substrate, a semiconductor fin over the substrate and between the first isolation structure and the second isolation structure, and a third isolation structure extending through the semiconductor fin and between the first isolation structure and the second isolation structure. A top surface of the semiconductor fin is above a top surface of the first isolation structure and a top surface of the second isolation structure. The third isolation structure includes a first dielectric material and a second dielectric material over the first dielectric material. An interface between the first dielectric material and the second dielectric material is below the top surface of the first isolation structure and the top surface of the second isolation structure.