A virtual memory manager for a multi-media engine allows individual media units to operate in their own virtual space in much the same way as a software program operating in virtual mode. The virtual memory controller performs address translation or mapping to the correct physical memory location (in local memory or system memory) and will also convert the data stream to or from a compressed format. In addition, the virtual memory controller provides a unified TLB (translation lookaside buffer) available to all media units. The TLB has four types of pointer entries which are controlled by two bits. The first bit controls whether the TLB entry is a direct map or a pointer to another translation table. the second bit controls whether the TLB entry is stored in a compressed format. The overall concept may allow dynamic load balancing between local media memory and system memory.