A stack including a silicon oxide layer, a germanium-containing layer, and a III-V compound semiconductor layer is formed over a substrate. An alternating stack of insulating layers and spacer material layers is formed over the III-V compound semiconductor layer. The spacer material layers are formed as, or are subsequently replaced with, electrically conductive layers. Memory openings are formed through the alternating stack and into the III-V compound semiconductor layer. Memory opening fill structures including a memory film and a vertical semiconductor channel are formed in the memory openings. The vertical semiconductor channels can include a III-V compound semiconductor channel material that is electrically connected to the III-V compound semiconductor layer. The substrate and at least a portion of the silicon oxide layer can be subsequently detached.