A computing device selects a feature set and hyperparameters for a machine learning model to predict a value for a characteristic in a scoring dataset. A number of training model iterations is determined. A unique evaluation pair is selected for each iteration that indicates a feature set selected from feature sets and a hyperparameter configuration selected from hyperparameter configurations. A machine learning model is trained using each unique evaluation pair. Each trained machine learning model is validated to compute a performance measure value. An estimation model is trained with the feature set, the hyperparameter configuration, and the performance measure value computed for unique evaluation pair. The trained estimation model is executed to compute the performance measure value for each unique evaluation pair. A final feature set and a final hyperparameter configuration are selected based on the computed performance measure value.