A method includes forming a hard mask over a substrate, patterning the hard mask to form a first plurality of trenches, and filling a dielectric material into the first plurality of trenches to form a plurality of dielectric regions. The hard mask is removed from between the plurality of dielectric regions, wherein a second plurality of trenches is left by the removed hard mask. An epitaxy step is performed to grow a semiconductor material in the second plurality of trenches.