Systems and techniques for an architecture for contextual memories in map representation for 3D reconstruction and navigation are described herein. In an example, a system for contextual memory mapping is adapted to receive a data set of physical world sensor readings. The system may be further adapted to generate voxel data from the data set, the voxel data includes voxel coordinates and a physical world occupancy indicator. The system may be further adapted to select a block of addresses in the memory to store the voxel data. The system may be further adapted to generate a hash map to map voxel coordinates to memory locations in the block of addresses, the voxel coordinates having a contextual relationship that is maintained by the hash map. The system may be further adapted to store the voxel data at memory addresses based on the hash map.