A method of forming a semiconductor device includes: forming a pattern having trenches on a semiconductor substrate; forming a semiconductor layer on the semiconductor device that fills the trenches; planarizing the semiconductor layer using a first planarization process without exposing the pattern; performing an epitaxy growth process on the first planarized semiconductor layer to form a crystalline semiconductor layer; and planarizing the crystalline semiconductor layer until the pattern is exposed to form a crystalline semiconductor pattern.