The present disclosure involves systems, software, and computer implemented methods for evaluating machine learning on remote datasets using confidentiality-preserving evaluation data. In response to determining that data of the remote customer dataset is of sufficient quality and quantity, feature data corresponding to a machine learning pipeline is generated. The remote customer dataset into one or more data partitions and for each partition, one or more baseline models and one or more machine learning models are trained using a machine learning library included in the remote customer database. Aggregate evaluation data is generated for each baseline model and each machine learning model that includes model debrief data and customer data statistics. In response to determining that the customer has enabled sharing of the aggregate evaluation data with a software provider who provided the remote customer database, the aggregate evaluation data is provided to the software provider.