Fabricating a three-dimensional memory device may include forming an alternating stack of insulating layers and sacrificial material layers over a substrate. Stepped surfaces are formed by patterning the alternating stack. Sacrificial pads are formed on physically exposed horizontal surfaces of the sacrificial material layers. A retro-stepped dielectric material portion is formed over the sacrificial pads. After memory stack structures extending through the alternating stack are formed, the sacrificial material layers and the sacrificial pads can be replaced with replacement material portions that include electrically conductive layers. The electrically conductive layers can be formed with thicker end portions. Contact via structures can be formed on the thicker end portions.