A system comprises a network of computers comprising a master computer and slave computers. For a machine learning problem that is partitioned into a number of correlated sub-problems, each master computer is configured to store tasks associated with the machine learning problem, and each of the slave computers is assigned one of the correlated sub-problems. Each slave computer is configured to store variables or parameters or both associated with the assigned one of the correlated sub-problems; obtain information about one or more tasks stored by the master computer without causing conflict with other slave computers with regard to the information; perform computations to update the obtained information and the variables or parameters or both of the assigned sub-problem; send the updated information to the master computer to update the information stored at the master computer; and store the updated variables or parameters or both of the assigned sub-problem.