A network device is configured to distribute audio streams and video streams among participant devices connected to the network device over a communication network. The network device collects respective audio streams from the participant devices, and ranks the participant devices based on the respective audio streams collected from the participant devices. Based on the ranking, the network device elects top-N participant devices among the participant devices as being associated with respective users who are active speakers, where N is one or more, and retrieves respective video streams only from the top-N participant devices. The network device receives from the participant devices respective requests for video streams from the top-N participant devices, and sends to the participant devices the video streams retrieved only from the top-N participant devices.