The technology disclosed relates to accelerating replay access to numerous streams via a channel aggregation hierarchy, muxing events from input buffers that receive event streams, into a physically aggregated stream in memory. A subscriber accesses events from the streams via the channel hierarchy; demuxing the physical aggregated stream and caching the events from the physical aggregated stream into event cache memory; caching references to selected events, which belong to active channels that have subscribers, in a caching structure including channel cache lists of the references, the caching structure including stream level channel cache lists, intermediate level channel cache lists, and, if the physical aggregate channel has subscribers, a physical aggregate channel cache list; and receiving a replay access request that specifies a channel and that specifies a starting point for the replay, and responsively delivering events from the event cache memory using a cache list corresponding to the specified channel.