Mapped redundant array of independent nodes employing mapped reliability groups for data storage

    公开(公告)号:US11847141B2

    公开(公告)日:2023-12-19

    申请号:US17152537

    申请日:2021-01-19

    IPC分类号: G06F16/28 G06F21/62 G06F16/21

    摘要: Storage of data according to mapped reliability groups (MRGs) of mapped redundant arrays of independent nodes (mapped clusters) is disclosed. Real reliability groups (RRGs) of a real cluster can provide compartmentalization that can reduce a probability of a data loss event occurring in a cluster storage system. MRGs for a mapped cluster employing the RRGs of the real cluster can provide additional compartmentalization that can further reduce a probability of a data loss event occurring. A data protection set can be stored according to a first group of MRGs and a redundant representation of the data protection set can be stored according to a second group of MRGs, wherein a network distance between the first and second group of MRGs is greater than a network distance between constituent MRGs of either the first or second group of MRGs.

    Cross-stream transactions in a streaming data storage system

    公开(公告)号:US11593030B2

    公开(公告)日:2023-02-28

    申请号:US17316842

    申请日:2021-05-11

    IPC分类号: G06F3/06

    摘要: The described technology is generally directed towards cross-stream transactions in a streaming data storage system, which allows a writer application to commit multiple events to distinct data streams in a single transaction. The system creates a cross-stream transaction for a writer application, and the writer application adds events to the cross-stream transaction, indicating which destination data stream(s) each event's data is to be appended. The system adds the event to a subordinate transaction created for each specified data stream. Upon committing the cross-stream transaction, the system coordinates the committing of the subordinate transactions to their respective data streams. Alternatively, an application writer can request that an event be committed to multiple writer-specified data streams via a cross-stream micro-transaction, in which the streaming data storage system obtains the appropriate segment store instances and segments for each data stream, and coordinates the direct appending of the event to the appropriate segments.

    Information-Unit Based Scaling of an Ordered Event Stream

    公开(公告)号:US20220342725A1

    公开(公告)日:2022-10-27

    申请号:US17237517

    申请日:2021-04-22

    IPC分类号: G06F9/54 G06F9/38 G06F11/34

    摘要: Scaling an ordered event stream (OES) based on an information-unit (IU) metric is disclosed. The IU metric can correspond to an amount of computing resources that can be consumed to access information embodied in event data of an event of the OES. In this regard, the amount of computing resources to access the data of the stream event itself can be distinct from an amount of computing resources employed to access information embodied in the data. As such, where an external application, e.g., a reader, a writer, etc., can connect to an OES data storage system, enabling the OES to be scaled in response to burdening of computing resources accessing event information, rather than merely event data, can aid in preservation of an ordering of events accessed from the OES.

    Selective instantiation of a storage service for a doubly mapped redundant array of independent nodes

    公开(公告)号:US11435957B2

    公开(公告)日:2022-09-06

    申请号:US16698096

    申请日:2019-11-27

    IPC分类号: G06F3/06 G06F12/10

    摘要: Selective instantiation of a mapped cluster storage service (MCSS) is disclosed for storage system supporting a doubly mapped redundant array of independent nodes environment. An instance of a MCSS can be run on computing resources of the storage system supporting a doubly mapped redundant array of independent nodes environment, wherein a real node of a real cluster comprises a real storage device that comprises real extents, and wherein extents of the storage system map to a mapped node of a mapped cluster. The instance of the MCSS can be run near to the highest number of real extents participating in a mapped node. This can reduce computing resource consumption associated with running the MCSS farther from the more concentrated real extents.

    Doubly mapped redundant array of independent nodes for data storage with high performance

    公开(公告)号:US11231860B2

    公开(公告)日:2022-01-25

    申请号:US16745855

    申请日:2020-01-17

    IPC分类号: G06F3/06

    摘要: The described technology is generally directed towards mapping doubly mapped storage clusters to resources of a real storage cluster in a way that provides high performance. In one aspect, the doubly mapped storage clusters are divided into logical columns, with each logical column corresponding to a doubly mapped node, and having a column height corresponding to a number of storage resources (e.g., disks multiplied by disk extents) managed by that doubly mapped node. The columns are logically positioned within a logical profile having dimensions of the real storage cluster. For example, the logical columns can be selected based on column height, and placed in the logical profile based on free disk extents of the nodes, greatest number of free disk extents first. Once logically positioned, the logical columns in the logical rectangle establish the mapping (e.g., embodied in a mapping table) that results in high performance.

    Watermarking of Events of an Ordered Event Stream

    公开(公告)号:US20210406273A1

    公开(公告)日:2021-12-30

    申请号:US16915762

    申请日:2020-06-29

    IPC分类号: G06F16/25 G06F16/23

    摘要: Watermarking of an ordered event stream (OES) is disclosed. Writing events to an OES can advance populating of the OES, which can be correspond to progress values. Progress values can enable a reader to know where it is in an OES. A reported progress value can be determined from the progress values based on an event being committed to the OES. Moreover, an effective progress value can be determined from the progress values and monitoring the OES for events with that defer being committed to the OES. A watermark for the OES can be determined from the effective progress value such that reading beyond the effective progress value is restricted. This can provide confidence that the reader has read out properly committed events and has not omitted any events having deferred commitment.

    Disaster recovery in a streaming data storage system

    公开(公告)号:US11210179B1

    公开(公告)日:2021-12-28

    申请号:US17154274

    申请日:2021-01-21

    IPC分类号: G06F11/14

    摘要: A streaming data storage system maintains a hierarchical structure of metadata in association with the data chunks of streams, in which the metadata facilitates recovery of the data streams if the streaming data storage system fails. In one implementation, the metadata comprises the pathnames and filenames of the chunks stored in a file storage system or object storage system, in which each pathname represents the epoch and segment of the chunks in the segment, and the chunks' filenames represent the relative ordering of the chunks in the segment. To recover the data stream, the epochs and their segments are recreated, and the segments are repopulated with references to their respective chunks. Once recovered, a new epoch is created with a number of active segments equal to the number of segments of the last recreated epoch, and event appends to the data stream can resume.

    Logical compaction of a degraded chunk in a geographically diverse data storage system

    公开(公告)号:US11119683B2

    公开(公告)日:2021-09-14

    申请号:US16228624

    申请日:2018-12-20

    IPC分类号: G06F3/06 G06F12/10

    摘要: Generating, in a geographically diverse storage system, a degraded convolved chunk that consumes less storage space than a convolved chunk that is not a degraded chunk is disclosed. The degraded convolved chunk can be generated at a third zone of the storage system and be based on a compressed representation of a first chunk from a first zone of the storage system and a second chunk from a second zone of the storage system, wherein the first chunk is a degraded chunk that comprises at least one non-relevant chunk fragment. In an embodiment, the compressed representation can be generated at the first zone. In another embodiment the compressed representation can be generated at the third zone. In an aspect, mapping data corresponding a logical compression of first chunk fragments to physical storage locations of the first chunk fragments can be employed in data operations of the storage system.

    DOUBLY MAPPED REDUNDANT ARRAY OF INDEPENDENT NODES FOR DATA STORAGE WITH HIGH PERFORMANCE

    公开(公告)号:US20210223969A1

    公开(公告)日:2021-07-22

    申请号:US16745855

    申请日:2020-01-17

    IPC分类号: G06F3/06

    摘要: The described technology is generally directed towards mapping doubly mapped storage clusters to resources of a real storage cluster in a way that provides high performance. In one aspect, the doubly mapped storage clusters are divided into logical columns, with each logical column corresponding to a doubly mapped node, and having a column height corresponding to a number of storage resources (e.g., disks multiplied by disk extents) managed by that doubly mapped node. The columns are logically positioned within a logical profile having dimensions of the real storage cluster. For example, the logical columns can be selected based on column height, and placed in the logical profile based on free disk extents of the nodes, greatest number of free disk extents first. Once logically positioned, the logical columns in the logical rectangle establish the mapping (e.g., embodied in a mapping table) that results in high performance.