Mapreduce implementation in an on-demand network code execution system and stream data processing system
Abstract:
Systems and methods are described for providing an implementation of the MapReduce programming model utilizing tasks executing on an on-demand code execution system, utilizing a stream data processing system as an intermediary between map and reduce function. A map task implementing a map function can process portions of a data set, to generate outputs associated with different values for a measured attribute of the data set. Executions of the map task can publish outputs to a data stream on the stream data processing system, which stream is configured to utilize the measured attribute as a partition key for the stream. Based on the partition key, the stream data processing system can divide the stream into sub-streams, each containing a relevant subset of the outputs. The on-demand code execution system can execute a reduce task to apply the reduce function to the outputs of each sub-stream, thereby completing the MapReduce process.
Information query
Patent Agency Ranking
0/0