One method disclosed includes removing at least a portion of a fin to thereby define a fin trench in a layer of insulating material, forming a substantially defect-free first layer of semiconductor material in the fin trench, forming a second layer of semiconductor material on an as-formed upper surface of the first layer of semiconductor material, forming an implant region at the interface between the first layer of semiconductor material and the substrate, performing an anneal process to induce defect formation in at least the first layer of semiconductor material, forming a third layer of semiconductor material on the second layer of semiconductor material, forming a layer of channel semiconductor material on the third layer of semiconductor material, and forming a gate structure around at least a portion of the channel semiconductor material.