An optimized service-based pipeline includes a resource manager that receives a request that includes a description of a workload from a workload initiator such as an application. The resource manager identifies runtime utilization metrics of a plurality of processing resources, where the plurality of processing resources includes at least a first graphics processing unit (GPU) and a second GPU. The resource manager determines, based on the utilization metrics and one or more policies, a workload allocation recommendation for the workload. Thus, the workload initiator can determine whether placing a workload on a particular processing resource is preferable based on runtime behavior of the system and policies established of the workload.