A memory system having a set of media, a set of resources, and a controller configured via firmware to use the set of resources in processing requests from a host system to store data in the media or retrieve data from the media. The memory system has a workload manager that analyzes activity records in an execution log for a time period where each of the activity records can indicate whether a processor of the controller is in an idle state during a time slot in the time period. The workload manager identifies idle time slots within the time period during which time slots one or more lightly-loaded processors in the plurality of processors are in the idle state, and adjusts a configuration of the controller to direct tasks from one or more heavily-loaded processors to the one or more lightly-loaded processors.