System and techniques for infrastructure managed workload distribution are described herein. An infrastructure processing unit (IPU) receives a workload that includes a workload definition. The workload definition includes stages of the workload and a performance expectation. The IPU provides the workload, for execution, to a processing unit of a compute node to which the IPU belongs. The IPU monitors execution of the workload to determine that a stage of the workload is performing outside of the performance expectation from the workload definition. In response, the IPU modifies the execution of the workload.