A method includes etching a hybrid substrate to form a recess extending into the hybrid substrate. The hybrid substrate includes a first semiconductor layer having a first surface orientation, a dielectric layer over the first semiconductor layer, and a second semiconductor layer having a second surface orientation different from the first surface orientation. After the etching, a top surface of the first semiconductor layer is exposed to the recess. A spacer is formed on a sidewall of the recess. The spacer contacts a sidewall of the dielectric layer and a sidewall of the second semiconductor layer. An epitaxy is performed to grow an epitaxy semiconductor region from the first semiconductor layer. The spacer is removed.