After formation of an alternating stack of insulating layers and sacrificial material layers, a memory opening can be formed through the alternating stack, which is subsequently filled with a columnar semiconductor pedestal portion and a memory stack structure. Breakage of the columnar semiconductor pedestal portion under mechanical stress can be avoided by growing a laterally protruding semiconductor portion by selective deposition of a semiconductor material after removal of the sacrificial material layers to form backside recesses. At least an outer portion of the laterally protruding semiconductor portion can be oxidized to form a tubular semiconductor oxide spacer. Electrically conductive layers can be formed in the backside recesses to provide word lines for a three-dimensional memory device.