One embodiment of the present invention is a technique for optimizing a task graph that specifies multiple tasks and the dependencies between the specified tasks. When optimizing the task graph, the optimization engine performs multiple iterations of runtime optimization operations on the task graph. At each iteration, an optimized task graph is generated based on a different task aggregation topology. The optimized task graph is then compiled and executed. Runtime statistics related to the execution are collected, and, in subsequent iterations, the task graph is further optimized based on the collected statistics. Once the optimization process is complete, the most optimal task graph topology that was identified during the process is used to generate an optimized task graph for execution.