A three-dimensional memory device includes an alternating stack of insulating layers and electrically conductive layers located over a substrate, memory stack structures extending through the alternating stack, where each of the memory stack structures contains a respective memory film and a respective vertical semiconductor channel, drain regions contacting an upper end of a respective one of the vertical semiconductor channels, first contact via structures directly contacting a first subset of the drain regions and each having a first horizontal cross-sectional area, and second contact via structures directly contacting a second subset of the drain regions and each having a second horizontal cross-sectional area that is greater than the first horizontal cross-sectional area.