A first blocking dielectric layer is formed in a memory opening through a stack of an alternating plurality of material layers and insulator layers. A spacer with a bottom opening is formed over the first blocking dielectric layer by deposition of a conformal material layer and an anisotropic etch. A horizontal portion of the first blocking dielectric layer at a bottom of the memory opening can be etched by an isotropic etch process that minimizes overetch into the substrate. An optional additional blocking dielectric layer, at least one charge storage element, a tunneling dielectric, and a semiconductor channel can be sequentially formed in the memory opening to provide a three-dimensional memory stack.