A model management system for a building, including one or more memory devices and one or more processors. The one or more memory devices are configured to store instructions to be executed on the one or more processors. The one or more processors are configured to determine whether chiller fault data exists in chiller data used to generate a plurality of chiller shutdown prediction models. The one or more processors are further configured to generate a first performance evaluation value for each of the plurality of chiller shutdown prediction models using a first evaluation technique in response to a determination that chiller fault data exists in the chiller data, and generate a second performance evaluation value for each of the plurality of chiller shutdown prediction models using a second evaluation technique in response to a determination that chiller fault data does not exist in the chiller data. The one or more processors are configured to select one of the plurality of chiller shutdown prediction models based on the first performance evaluation in response to the determination that chiller fault data exists in the chiller data, and select one of the plurality of chiller shutdown prediction models based on the second performance evaluation in response to the determination that chiller fault data does not exist in the chiller data.