Word line switches in a word line decoder circuitry for a three-dimensional memory device can be formed as vertical field effect transistors overlying contact via structures to the electrically conductive layers for word lines. Via cavities in a dielectric material portion overlying stepped surfaces of the electrically conductive layers can be filled with a conductive material and recessed to form contact via structures. After forming lower active regions in the recesses, gate electrodes can be formed and patterned to form openings in areas overlying the contact via structures. Gate dielectrics can be formed on the sidewalls of the openings, and transistor channels can be formed inside the openings of the gate electrodes. Upper active regions can be formed over the transistor channels.