A semiconductor device with isolation structures of different dielectric constants and a method of fabricating the same are disclosed. The semiconductor device includes fin structures with first and second fin portions disposed on first and second device areas on a substrate and first and second pair of gate structures disposed on the first and second fin portions. The second pair of gate structures is electrically isolated from the first pair of gate structures. The semiconductor device further includes a first isolation structure interposed between the first pair of gate structures and a second isolation structure interposed between the second pair of gate structures. The first isolation structure includes a first nitride liner and a first oxide fill layer. The second isolation structure includes a second nitride liner and a second oxide fill layer. The second nitride liner is thicker than the first nitride liner.