A semiconductor device structure, along with methods of forming such, are described. The structure includes first and second dielectric features and a first semiconductor layer disposed between the first and second dielectric features. The structure further includes an isolation layer disposed between the first and second dielectric features, and the isolation layer is in contact with the first and second dielectric features. The first semiconductor layer is disposed over the isolation layer. The structure further includes a gate dielectric layer disposed over the isolation layer and a gate electrode layer disposed over the gate dielectric layer. The gate electrode layer has an end extending to a level between a first plane defined by a first surface of the first semiconductor layer and a second plane defined by a second surface opposite the first surface.