A computer-implemented method, a computer program product, and a computer system for parallel cross validation in collaborative machine learning. A server groups local models into groups. In each group, each local device uses its local data to validate accuracies of the local models and sends a validation result to a group leader or the server. The group leader or the server selects groups whose variances of the accuracies are not below a predetermined variance threshold. In each selected group, the group leader or the server compares an accuracy of each local model with an average value of the accuracies and randomly selects one or more local models whose accuracies do not exceed a predetermined accuracy threshold. The server obtains weight parameters of selected local models and updates the global model based on the weight parameters.