An adaptive scaler switching system may implement multiple scalers including both a software scaler and a hardware scaler, and a controller that may manage the switch between scalers by considering the real-time constraints of the system and the available system resources. Information about the availability of system resources may be received in real-time, for example the controller may receive information about the system thermal status, the timing requirements for processing the video data, the quality of the scaled data, and any other relevant system statistics that may affect the scaler switch decision. According to an embodiment, the system may maintain statistics in a table, and update the table information as necessary.