Techniques disclosed herein describe a data retrieval process for storing management data from a network environment using an efficient polling-based approach without sacrificing the real-time aspect of data retrieval that a push model provides. The process is highly scalable and reliable, and is useful in Network Address Translation (NAT) environments because the number of holes in a NAT firewall is minimized. The data retrieval component uses intelligent polling to retrieve data and store the data in a central database without overloading the system. A data retrieval component polls a data collection agent for a dataset. The data collection agent transfers the data set and a poll indication of when a subsequent data set will be ready for transfer. Subsequent polling is based on a processor load at a data retrieval component.