A method and system for automatically determining an optimal re-training interval for a fault diagnoser based on online monitoring of the performance of a classifier. The classifier generates a soft measure of membership in association with a class based on a training data. The output of the classifier can be utilized to assign a label to new data and then the members associated with each class can be clustered into one or more core members and potential outliers. A statistical measure can be utilized to determine if the distribution of the outliers is sufficiently different than the core members after enough outliers have been accumulated. If the outliers are different with respect to the core members, then the diagnoser can be re-trained; otherwise, the output of the classifier can be fed to the fault diagnoser.