Memory openings and support openings can be formed through an alternating stack of insulating layers and sacrificial material layers. A set of dielectric layers and at least one semiconductor material layer can be sequentially deposited in each of the memory openings and the support openings. The at least one semiconductor material layer is removed from inside the support openings, while the at least one semiconductor material layer is not removed from inside the memory openings. Memory stack structures and support pillar structures are formed in the memory openings and the support openings, respectively. The sacrificial material layers are replaced with electrically conductive layers. Removal of the at least one semiconductor material layer from the support pillar structures reduces or eliminates leakage current through the support pillar structures.