Invention Grant
- Patent Title: Mapreduce implementation in an on-demand network code execution system and stream data processing system
-
Application No.: US16144997Application Date: 2018-09-27
-
Publication No.: US11243953B2Publication Date: 2022-02-08
- Inventor: Timothy Allen Wagner , Hans-Philipp Anton Hussels , Marc John Brooker
- Applicant: Amazon Technologies, Inc.
- Applicant Address: US WA Seattle
- Assignee: Amazon Technologies, Inc.
- Current Assignee: Amazon Technologies, Inc.
- Current Assignee Address: US WA Seattle
- Agency: Knobbe, Martens, Olson & Bear, LLP
- Main IPC: G06F16/24
- IPC: G06F16/24 ; G06F16/2455 ; G06F16/951

Abstract:
Systems and methods are described for providing an implementation of the MapReduce programming model utilizing tasks executing on an on-demand code execution system, utilizing a stream data processing system as an intermediary between map and reduce function. A map task implementing a map function can process portions of a data set, to generate outputs associated with different values for a measured attribute of the data set. Executions of the map task can publish outputs to a data stream on the stream data processing system, which stream is configured to utilize the measured attribute as a partition key for the stream. Based on the partition key, the stream data processing system can divide the stream into sub-streams, each containing a relevant subset of the outputs. The on-demand code execution system can execute a reduce task to apply the reduce function to the outputs of each sub-stream, thereby completing the MapReduce process.
Information query