Embodiments herein describe a multi-port memory system that includes one or more single port memories (e.g., a memory that can perform only one read or one write at any given time, referred to as a 1W or 1R memory). That is, the multi-port memory system can perform multiple read and writes in parallel (e.g., 1R/1W, 1R/3W, 2R/2W, 3R/1W, etc.) even though the memory in the system can only perform one read or one write at any given time. The advantage of doing so is a reduction in area and power.