A semiconductor device includes a substrate, a semiconductor fin, an isolation plug, and an isolation structure. The semiconductor fin is over the substrate. The isolation plug is over the substrate and adjacent to an end of the semiconductor fin. The isolation structure is over the substrate and adjacent to sidewalls of the semiconductor fin and the isolation plug. A top surface of the isolation structure is in a position lower than a top surface of the isolation plug.