A method of forming a resistive memory device includes forming an alternating stack of insulating layers and sacrificial material layers that extend along a first horizontal direction over a substrate, forming a laterally alternating sequence of vertical conductive lines and dielectric pillar structures that alternate along the first horizontal direction on sidewalls of the alternating stack, forming lateral recesses by removing the sacrificial material layers selective to the insulating layers, selectively growing resistive memory material portions from physically exposed surfaces of the vertical conductive lines in the lateral recesses, and forming electrically conductive layers over the resistive memory material portions in the lateral recesses.