To reduce latency in a network device that buffer packets in different queues based on class of service, packets received from a network are stored in physical queues according to a class of service associated with the packets and a class of service associated with each of the physical queues. The physical queues are scheduled based quality of service requirements of their associated class of service. The physical queues are shadowed by virtual queues, and whether congestion exists in at least one of the virtual queues is determined. Packets departing from at least one of the physical queues are marked when congestion exists in at least one of the virtual queues. The service rate of the virtual queues is set to be less than or equal to a port link rate of the network device.