In one embodiment, a supervisory service for a software-defined wide area network (SD-WAN) uses a plurality of different decision thresholds for a machine learning-based classifier, to predict tunnel failures of a tunnel in the SD-WAN. The supervisory service captures performance data indicative of performance of the classifier when using the different decision thresholds. The supervisory service selects, based on the captured performance data, a particular decision threshold for the classifier, in an attempt to optimize the performance of the classifier. The supervisory service uses the selected decision threshold for the classifier, to predict a tunnel failure of the tunnel.