In one embodiment, a network assurance service that monitors a network detects a behavioral anomaly in the network using an anomaly detector that compares an anomaly detection threshold to a target value calculated based on a first set of one or more measurements from the network. The service uses an explanation model to predict when the anomaly detector will detect anomalies. The explanation model takes as input a second set of one or more measurements from the network that differs from the first set. The service determines that the detected anomaly is explainable, based on the explanation model correctly predicting the detection of the anomaly by the anomaly detector. The service provides an anomaly detection alert for the detected anomaly to a user interface, based on the detected anomaly being explainable. The anomaly detection alert indicates at least one measurement from the second set as an explanation for the anomaly.