A three-dimensional memory is formed as an array of memory elements that are formed across multiple layers of planes positioned at different distances above a semiconductor substrate. The memory elements reversibly change a level of electrical conductance in response to a voltage difference being applied across them. The three-dimensional array includes a two-dimensional array of pillar lines acting as local vertical bit lines through the multiple layers of planes which together with arrays of word lines on each plane are used to access the memory elements. The three-dimensional memory is formed over a CMOS substrate with an intermediate pillar select layer. The pillar select layer is formed with a plurality of pillar select devices which are switching transistors formed outside the CMOS and serve to switch selected rows of pillar lines to corresponding metal lines on the substrate.