Methods and systems for configuring a machine learning model include selecting a head from a set of stored heads, responsive to an input, to implement a layer in a transformer machine learning model. The selected head is copied from persistent storage to active memory. The layer in the transformer machine learning model is executed on the input using the selected head to generate an output. An action is performed responsive to the output.