A neural network models fragmenting method, system, and computer program product include recursively factoring out common prefixes of models, constructing a hierarchy of decomposed model fragments based on the factoring, and grouping the constructed hierarchy for deployment.