A shallow trench isolation (STI) structure and a method of forming the STI structure. The STI structure defines an active region formed with a recess channel transistor. The STI structure includes a STI trench has a laterally curved rounding portion on the bottom of the recess channel trench. In order to form the STI trench with the rounding portion, a semiconductor substrate is selectively and anisotropically dry etched to form the trench. Then, the semiconductor substrate is isotropically etched around the bottom height of the recess channel trench to form the rounding portion, and then further anisotropically dry etched, thereby forming the STI trench. After an insulating layer that fill the STI trench is formed on the resultant structure, an upper surface of the resultant structure is planarized to expose a surface of the semiconductor substrate.