Embodiments are directed to homomorphic encryption for machine learning and neural networks using high-throughput Chinese remainder theorem (CRT) evaluation. An embodiment of an apparatus includes a hardware accelerator to receive a ciphertext generated by homomorphic encryption (HE) for evaluation, decompose coefficients of the ciphertext into a set of decomposed coefficients, multiply the decomposed coefficients using a set of smaller modulus determined based on a larger modulus, and convert results of the multiplying back to an original form corresponding to the larger modulus by performing a reverse Chinese remainder theorem (CRT) transform on the results of multiplying the decomposed coefficients.