In one embodiment, a device in a network monitors performance data for a first predictive model. The first predictive model is used to make proactive decisions in the network. The device maintains a supervisory model based on the monitored performance data for the first predictive model. The device identifies a time period during which the supervisory model predicts that the first predictive model will perform poorly. The device causes a switchover from the first predictive model to a second predictive model at a point in time associated with the time period, in response to identifying the time period.