In one embodiment, a supervisory service for one or more networks receives telemetry data samples from a plurality of networking devices in the one or more networks. The service trains a failure prediction model to predict failures in the one or more networks, using a training dataset comprising the received telemetry data samples. The service assesses performance of the failure prediction model. The service trains, based on the assessed performance of the failure prediction model, a machine learning-based classification model to determine whether a networking device should send a particular telemetry data sample to the service. The service sends the machine learning-based classifier to one or more of the plurality of networking devices, to control which telemetry data samples the one or more networking devices send to the supervisory service.