公开(公告)号：US11194506B1

公开(公告)日：2021-12-07

申请号：US16940461

申请日：2020-07-28

Applicant: NetApp, Inc.

Inventor： Alyssa Proulx ,  Mark David Olson

IPC: G06F3/06

Abstract: A system, method, and machine-readable storage medium for determining an amount of unique data in a distributed storage system are provided. In some embodiments, a combined efficiency set for a first data set stored in the distributed storage system, such as at a volume, may be generated. The first data set may include a first subset of data and a second subset of data in the distributed storage system. Additionally, a set of efficiency sets for the first subset of data may be generated. A set difference based on the combined efficiency set and the set of efficiency sets may be computed. An amount of memory used for storing unique data of the second subset of data may be estimated based on the set difference. The unique data may be present in the second subset of data but absent from the first subset of data.

公开(公告)号：US12189981B2

公开(公告)日：2025-01-07

申请号：US18161391

申请日：2023-01-30

Applicant: NetApp, Inc.

Inventor： Alyssa Proulx ,  Mark David Olson

IPC: G06F3/06

Abstract: A system, method, and machine-readable storage medium for determining an amount of unique data in a distributed storage system are provided. In some embodiments, a combined efficiency set for a first data set stored in the distributed storage system, such as at a volume, may be generated. The first data set may include a first subset of data and a second subset of data in the distributed storage system. Additionally, a set of efficiency sets for the first subset of data may be generated. A set difference based on the combined efficiency set and the set of efficiency sets may be computed. An amount of memory used for storing unique data of the second subset of data may be estimated based on the set difference. The unique data may be present in the second subset of data but absent from the first subset of data.

公开(公告)号：US20230176773A1

公开(公告)日：2023-06-08

申请号：US18161391

申请日：2023-01-30

Applicant: NetApp, Inc.

Inventor： Alyssa Proulx ,  Mark David Olson

IPC: G06F3/06

CPC classification number: G06F3/0655 ,  G06F3/067 ,  G06F3/0604

Abstract: A system, method, and machine-readable storage medium for determining an amount of unique data in a distributed storage system are provided. In some embodiments, a combined efficiency set for a first data set stored in the distributed storage system, such as at a volume, may be generated. The first data set may include a first subset of data and a second subset of data in the distributed storage system. Additionally, a set of efficiency sets for the first subset of data may be generated. A set difference based on the combined efficiency set and the set of efficiency sets may be computed. An amount of memory used for storing unique data of the second subset of data may be estimated based on the set difference. The unique data may be present in the second subset of data but absent from the first subset of data.

公开(公告)号：US20230145784A1

公开(公告)日：2023-05-11

申请号：US17680484

申请日：2022-02-25

Applicant: NetApp, Inc.

Inventor： Wei Sun ,  Mark David Olson ,  Anil Paul Thoppil

IPC: G06F12/02 ,  G06F16/22

CPC classification number: G06F12/0253 ,  G06F16/2246 ,  G06F16/2272

Abstract: Systems and methods are described for a streamlined garbage collection process during which data integrity checking is also performed for a distributed key-value (KV) store utilized by a distributed storage management system. According to one embodiment, by making use of full or truncated block IDs (rather than an intermediate probabilistic data structure, such as a Bloom filter) for garbage collection, data integrity checking can be performed concurrently almost for free. During garbage collection, a block ID compare list is compared to block IDs within the distributed KV store. If a particular block ID is present in the distributed KV store but is missing from the block ID compare list, the corresponding data block represents garbage to be collected. If the particular block ID is present in the block ID compare list but missing from the distributed KV store, a data integrity error has been identified.

公开(公告)号：US11567694B2

公开(公告)日：2023-01-31

申请号：US17457117

申请日：2021-12-01

Applicant: NetApp, Inc.

Inventor： Alyssa Proulx ,  Mark David Olson

IPC: G06F3/06

Abstract: A system, method, and machine-readable storage medium for determining an amount of unique data in a distributed storage system are provided. In some embodiments, a combined efficiency set for a first data set stored in the distributed storage system, such as at a volume, may be generated. The first data set may include a first subset of data and a second subset of data in the distributed storage system. Additionally, a set of efficiency sets for the first subset of data may be generated. A set difference based on the combined efficiency set and the set of efficiency sets may be computed. An amount of memory used for storing unique data of the second subset of data may be estimated based on the set difference. The unique data may be present in the second subset of data but absent from the first subset of data.

公开(公告)号：US20240385959A1

公开(公告)日：2024-11-21

申请号：US18786848

申请日：2024-07-29

Applicant: NetApp, Inc.

Inventor： Wei Sun ,  Mark David Olson ,  Anil Paul Thoppil

IPC: G06F12/02 ,  G06F16/22

Abstract: Systems and methods are described for a streamlined garbage collection process during which data integrity checking is also performed for a distributed key-value (KV) store utilized by a distributed storage system. According to one embodiment, by making use of full or truncated block IDs (rather than an intermediate probabilistic data structure, such as a Bloom filter) for garbage collection, data integrity checking can be performed concurrently almost for free. During garbage collection, a block ID compare list may be compared to block IDs within the distributed KV store. If a particular block ID is present in the distributed KV store but is missing from the block ID compare list, the corresponding data block represents garbage to be collected. If the particular block ID is present in the block ID compare list but missing from the distributed KV store, a data integrity error has been identified.

公开(公告)号：US12066933B2

公开(公告)日：2024-08-20

申请号：US17680484

申请日：2022-02-25

Applicant: NetApp, Inc.

Inventor： Wei Sun ,  Mark David Olson ,  Anil Paul Thoppil

IPC: G06F12/02 ,  G06F16/22

CPC classification number: G06F12/0253 ,  G06F16/2246 ,  G06F16/2272

Abstract: Systems and methods are described for a streamlined garbage collection process during which data integrity checking is also performed for a distributed key-value (KV) store utilized by a distributed storage management system. According to one embodiment, by making use of full or truncated block IDs (rather than an intermediate probabilistic data structure, such as a Bloom filter) for garbage collection, data integrity checking can be performed concurrently almost for free. During garbage collection, a block ID compare list is compared to block IDs within the distributed KV store. If a particular block ID is present in the distributed KV store but is missing from the block ID compare list, the corresponding data block represents garbage to be collected. If the particular block ID is present in the block ID compare list but missing from the distributed KV store, a data integrity error has been identified.

公开(公告)号：US20220083262A1

公开(公告)日：2022-03-17

申请号：US17457117

申请日：2021-12-01

Applicant: NetApp, Inc.

Inventor： Alyssa Proulx ,  Mark David Olson

IPC: G06F3/06

Abstract: A system, method, and machine-readable storage medium for determining an amount of unique data in a distributed storage system are provided. In some embodiments, a combined efficiency set for a first data set stored in the distributed storage system, such as at a volume, may be generated. The first data set may include a first subset of data and a second subset of data in the distributed storage system. Additionally, a set of efficiency sets for the first subset of data may be generated. A set difference based on the combined efficiency set and the set of efficiency sets may be computed. An amount of memory used for storing unique data of the second subset of data may be estimated based on the set difference. The unique data may be present in the second subset of data but absent from the first subset of data.