A method of manufacturing a structure includes forming an alternating stack of insulating layers (42) and spacer material layers (32) over a substrate (9), dividing the alternating stack into a first alternating stack (100, 300) and a second alternating stack (200), the first alternating stack having first stepped surfaces and the second alternating stack having second stepped surfaces, forming at least one memory stack structure through the first alternating stack (100), each of the at least one memory stack structure including charge storage regions, a tunneling dielectric, and a semiconductor channel, replacing portions of the insulating layers in the first alternating stack with electrically conductive layers (46) while leaving intact portions of the insulating layers (42) in the second alternating stack, and forming a contact via structure (84) through the second alternating stack to contact a peripheral semiconductor device under the second stack.