Peripheral devices for a three-dimensional memory device can be formed over an array of memory stack structures to increase areal efficiency of a semiconductor chip. First contact via structures and first metal lines are formed over an array of memory stack structures and an alternating stack of insulating layers and electrically conductive layers. A semiconductor material layer including a single crystalline semiconductor material or a polycrystalline semiconductor material is formed over first metal lines. After formation of semiconductor devices on or in the semiconductor material layer, metal interconnect structures including second metal lines and additional conductive via structures are formed to electrically connect nodes of the semiconductor devices to respective first metal lines and to memory devices underneath.