An alternating stack of insulating layers and spacer material layers can be formed over a substrate. The spacer material layers may be formed as, or may be subsequently replaced with, electrically conductive layers. A memory opening can be formed through the alternating stack, and annular lateral recesses are formed at levels of the insulating layers. Metal portions are formed in the annular lateral recesses, and a semiconductor material layer is deposited over the metal portions. Metal-semiconductor alloy portions are formed by performing an anneal process, and are subsequently removed by performing a selective etch process. Remaining portions of the semiconductor material layer include a vertical stack of semiconductor material portions, which may be optionally converted, partly or fully, into silicon nitride material portions. The semiconductor material portions and/or the silicon nitride material portions can be employed as discrete charge storage elements.