In one embodiment, a network assurance service that monitors a network detects anomalies in the network by applying one or more machine learning-based anomaly detectors to telemetry data from the network. The network assurance service receives ranking feedback from a plurality of anomaly rankers regarding relevancy of the detected anomalies. The network assurance service calculates a rescaling factor and quantile parameter by applying an objective function to the ranking feedback, in order to optimize the rescaling factor and quantile parameter of the one or more anomaly detectors. The network assurance service adjusts the rescaling factor and quantile parameter of the one or more anomaly detectors using the calculated rescaling factor and quantile parameter.