Frame based streaming data is controlled through a reconfigurable graph of processing modules. A client specifies overall goals for the graph. A graph manager constructs the graph as a sequence of interconnected modules for processing the data, in response to the capabilities of modules within the graph and the overall goals, and divides the graph into time domains each having one or more modules, pipes each having one or more modules, and a control mechanism is used that eliminates components unnecessary to the overall operation of the graph and that provides synchronization between time domains using time translation tables or timing correlation tables. The graph manager adaptively controls graphs that have a low latency requirement.