A semiconductor structure includes an alternating stack of insulating layers and electrically conductive layers, a memory opening vertically extending through the alternating stack, and a memory opening fill structure located in the memory opening and including a vertical semiconductor channel, a memory film in contact with the vertical semiconductor channel, and a vertical stack of tubular dielectric spacers laterally surrounding the memory film. The tubular dielectric spacers may include tubular graded silicon oxynitride portions having a composition gradient such that an atomic concentration of nitrogen decreases with a lateral distance from an outer sidewall of the memory film, or may include tubular composite dielectric spacers including a respective tubular silicon oxide spacer and a respective tubular dielectric metal oxide spacer. Each of the electrically conductive layers has a hammerhead-shaped vertical cross-sectional profile.