A scalable server architecture for use in implementing scaled media servers capable of simultaneous real-time data stream retrieval for large numbers of subscribers. A scalable server includes a plurality of stream pumping engines each accessing a particular storage device of a storage subsystem, and a server processor which receives retrieval requests from subscribers and directs the stream pumping engines to retrieve the requested data streams. Each of the stream pumping engines may include a storage controller coupled to its corresponding storage device for directing retrieval of the requested stream therefrom, a network controller for supplying the retrieved stream to a client network, and a processor for directing the operation of the storage and network controllers. Each of the stream pumping engines may also include a shared memory accessible by the corresponding stream pumping engine processor and the server processor. The shared memory facilitates communication with other stream pumping engines via the server processor and server system bus. A scaled media server may be implemented by cross-connecting several scalable servers with a plurality of stream multiplexers. Each of the stream multiplexers can include a separate packet input unit for processing the packets of each media stream such that two distinct levels of transmission priority are provided and quality of server restrictions are satisfied for all streams.