Invention Application
- Patent Title: Scalable Exactly-Once Data Processing Using Transactional Streaming Writes
-
Application No.: US17085576Application Date: 2020-10-30
-
Publication No.: US20220138071A1Publication Date: 2022-05-05
- Inventor: Pavan Edara , Reuven Lax , Yi Yang , Gurpreet Singh Nanda
- Applicant: Google LLC
- Applicant Address: US CA Mountain View
- Assignee: Google LLC
- Current Assignee: Google LLC
- Current Assignee Address: US CA Mountain View
- Main IPC: G06F11/30
- IPC: G06F11/30 ; G06F11/07 ; G06F12/02 ; G06F9/30 ; G06F9/46

Abstract:
A method for processing data exactly once using transactional stream writes includes receiving, from a client, a batch of data blocks for storage on memory hardware in communication with the data processing hardware. The batch of data blocks is associated with a corresponding sequence number and represents a number of rows of a table stored on the memory hardware. The method also includes partitioning the batch of data blocks into a plurality of sub-batches of data blocks. For each sub-batch of data blocks, the method further includes assigning the sub-batch of data blocks to a buffered stream; writing, using the assigned buffered stream, the sub-batch of data blocks to the memory hardware; updating a storage log with an intent to commit the sub-batch of data blocks using the assigned buffered stream; and committing the sub-batch of data blocks to the memory hardware.
Public/Granted literature
- US11573876B2 Scalable exactly-once data processing using transactional streaming writes Public/Granted day:2023-02-07
Information query