An alternating stack of insulating layers and sacrificial material layers is formed over a substrate. A memory opening is formed through the alternating stack. A memory film including a silicon nitride layer and a tunneling dielectric layer is formed in the memory opening, and an opening is formed through the memory film. A chemical oxide layer is formed on a physically exposed surface of an underlying semiconductor material portion. A silicon nitride ring can be formed by selectively growing a silicon nitride material from an annular silicon nitride layer portion of the silicon nitride layer while suppressing deposition of the silicon nitride material on the tunneling dielectric layer and on the chemical oxide layer. A vertical semiconductor channel can be formed by depositing a continuous semiconductor material layer on the underlying semiconductor material portion and the tunneling dielectric layer and on the silicon nitride ring.