公开(公告)号：US09298736B2

公开(公告)日：2016-03-29

申请号：US14293966

申请日：2014-06-02

Applicant: Google Inc.

Inventor： Yonatan Zunger ,  Alexandre Drobychev ,  Alexander Kesselman ,  Rebekah C. Vickrey ,  Frank C. Dachille ,  George Datuashvili

IPC: G06F17/00 ,  G06F7/00 ,  G06F17/30 ,  G06F13/10 ,  G06G7/62 ,  H04L29/08

CPC classification number: G06F17/30215 ,  G06F11/2094 ,  G06F11/3457 ,  H04L67/1095 ,  H04L67/1097

Abstract: A method allocates object replicas in a distributed storage system. The method identifies a plurality of objects in the distributed storage system. Each object has an associated storage policy that specifies a target number of object replicas stored at distinct instances of the distributed storage system. The method identifies an object of the plurality of objects whose number of object replicas exceeds the target number of object replicas specified by the storage policy associated with the object. The method selects a first replica of the object for removal based on last access times for replicas of the object, and transmits a request to a first instance of the distributed storage system that stores the first replica. The request instructs the first instance to remove the first replica of the object.

Abstract translation: 一种方法在分布式存储系统中分配对象副本。 该方法识别分布式存储系统中的多个对象。 每个对象具有关联的存储策略，其指定存储在分布式存储系统的不同实例处的对象副本的目标数量。 该方法识别多个对象的对象，其对象副本的数量超过与对象相关联的存储策略指定的对象副本的目标数量。 该方法基于对象的副本的最后访问时间选择要删除的对象的第一副本，并将请求发送到存储第一副本的分布式存储系统的第一实例。 请求指示第一个实例删除对象的第一个副本。

公开(公告)号：US09747322B2

公开(公告)日：2017-08-29

申请号：US14506530

申请日：2014-10-03

Applicant: GOOGLE INC.

Inventor： Alexandre Drobychev ,  Alexander Kesselman ,  Rebekah C. Vickrey ,  Frank C. Dachille ,  George Datuashvili

IPC: G06F7/00 ,  G06F17/00 ,  G06F17/30 ,  G06F3/06

CPC classification number: G06F17/30345 ,  G06F3/065 ,  G06F17/30082 ,  G06F17/30554 ,  G06F17/30575

Abstract: A distributed storage system has multiple instances. There is a plurality of local instances, and at least some of the local instances are at physically distinct geographic locations. Each local instance is configured to store data for a non-empty set of blobs in a plurality of data stores having a plurality of distinct data store types. In addition, each local instance stores metadata for the respective set of blobs in a metadata store distinct from the data stores. There is also a plurality of global instances. Each global instance is configured to store data for zero or more blobs in zero or more data stores and store metadata for all blobs stored at any local or global instance. The system selects one global instance to run a replication module that replicates blobs between instances according to blob policies. Some systems also include dynamic replication based on user needs.

公开(公告)号：US20160275125A1

公开(公告)日：2016-09-22

申请号：US15144700

申请日：2016-05-02

Applicant: GOOGLE INC.

Inventor： Alexandre Drobychev ,  Alexander Kesselman ,  Rebekah C. Vickrey ,  Frank C. Dachille ,  George Datuashvili

IPC: G06F17/30

CPC classification number: G06F16/23 ,  G06F3/065 ,  G06F16/122 ,  G06F16/248 ,  G06F16/27

Abstract: A distributed storage system stores data for files. A first blob (binary large object) of data is received. The first blob is split into one or more first chunks of data. Content fingerprints for the first chunks of data are computed. The first chunks of data are stored in a chunk store while and their content fingerprints are stored in a store distinct from the chunk store. A second blob of data is received. The second blob is split into one or more second chunks of data. Content fingerprints for the second chunks of data are computed. Then for a second chunk of data whose content fingerprint matches a content fingerprint of a first chunk of data, a second reference to the corresponding first chunk of data that has a matching content fingerprint is stored, but the second chunk of data is not stored.

Abstract translation: 分布式存储系统存储文件数据。 接收到第一个blob（二进制大对象）数据。 第一个blob被分成一个或多个第一批数据。 计算第一批数据的内容指纹。 第一批数据被存储在块存储器中，并且它们的内容指纹被存储在与块存储器不同的存储器中。 接收第二个数据块。 第二个Blob被分成一个或多个第二个数据块。 计算第二批数据的内容指纹。 然后，对于其内容指纹与第一组数据的内容指纹匹配的第二数据块，存储具有匹配内容指纹的对应的第一数据块的第二参考，但不存储第二组数据。

公开(公告)号：US09317524B2

公开(公告)日：2016-04-19

申请号：US14538716

申请日：2014-11-11

Applicant: GOOGLE INC.

Inventor： Yonatan Zunger ,  Alexandre Drobychev ,  Alexander Kesselman ,  Rebekah C. Vickrey ,  Frank C. Dachille ,  George Datuashvili

IPC: G06F17/00 ,  G06F7/00 ,  G06F17/30 ,  G06F13/10 ,  G06G7/62 ,  H04L29/08

CPC classification number: G06F17/30215 ,  G06F11/2094 ,  G06F11/3457 ,  H04L67/1095 ,  H04L67/1097

Abstract: A system and method for generating replication requests for objects in a distributed storage system is provided. For a respective object in a distributed storage system the following is performed. Replication policies for the object that have not been satisfied are determined. Replication requests are ranked for the object whose replication policies have not been satisfied based on a number of replicas of the object that need to be created in order to satisfy the replication policies for the object. Replication requests are generated for the object based at least in part on the replication policies for the object that have not been satisfied and on a current state of the distributed storage system. At least a subset of the replication requests for the objects in the distributed storage system are distributed to respective instances of the distributed storage system corresponding to the replication requests for execution.

Abstract translation: 提供了一种用于在分布式存储系统中生成对象的复制请求的系统和方法。 对于分布式存储系统中的相应对象，执行以下操作。 确定尚未满足的对象的复制策略。 基于需要创建的对象的副本的数量，为了满足对象的复制策略，复制请求被排序为其复制策略尚未满足的对象。 至少部分地基于对于尚未满足的对象和分布式存储系统的当前状态的复制策略为该对象生成复制请求。 对分布式存储系统中的对象的复制请求的至少一个子集分配到与复制请求执行相对应的分布式存储系统的相应实例。

公开(公告)号：US20150142743A1

公开(公告)日：2015-05-21

申请号：US14538716

申请日：2014-11-11

Applicant: GOOGLE INC.

Inventor： Yonatan Zunger ,  Alexandre Drobychev ,  Alexander Kesselman ,  Rebekah C. Vickrey ,  Frank C. Dachille ,  George Datuashvili

IPC: G06F17/30

CPC classification number: G06F17/30215 ,  G06F11/2094 ,  G06F11/3457 ,  H04L67/1095 ,  H04L67/1097

Abstract: A system and method for generating replication requests for objects in a distributed storage system is provided. For a respective object in a distributed storage system the following is performed. Replication policies for the object that have not been satisfied are determined. Replication requests are ranked for the object whose replication policies have not been satisfied based on a number of replicas of the object that need to be created in order to satisfy the replication policies for the object. Replication requests are generated for the object based at least in part on the replication policies for the object that have not been satisfied and on a current state of the distributed storage system. At least a subset of the replication requests for the objects in the distributed storage system are distributed to respective instances of the distributed storage system corresponding to the replication requests for execution.

Abstract translation: 提供了一种用于在分布式存储系统中生成对象的复制请求的系统和方法。 对于分布式存储系统中的相应对象，执行以下操作。 确定尚未满足的对象的复制策略。 基于需要创建的对象的副本的数量，为了满足对象的复制策略，复制请求被排序为其复制策略尚未满足的对象。 至少部分地基于对于尚未满足的对象和分布式存储系统的当前状态的复制策略为该对象生成复制请求。 对分布式存储系统中的对象的复制请求的至少一个子集分配到与复制请求执行相对应的分布式存储系统的相应实例。

公开(公告)号：US20150026128A1

公开(公告)日：2015-01-22

申请号：US14506530

申请日：2014-10-03

Applicant: GOOGLE INC.

Inventor： Alexandre Drobychev ,  Alexander Kesselman ,  Rebekah C. Vickrey ,  Frank C. Dachille ,  George Datuashvili

IPC: G06F17/30

CPC classification number: G06F17/30345 ,  G06F3/065 ,  G06F17/30082 ,  G06F17/30554 ,  G06F17/30575

Abstract: A distributed storage system has multiple instances. There is a plurality of local instances, and at least some of the local instances are at physically distinct geographic locations. Each local instance is configured to store data for a non-empty set of blobs in a plurality of data stores having a plurality of distinct data store types. In addition, each local instance stores metadata for the respective set of blobs in a metadata store distinct from the data stores. There is also a plurality of global instances. Each global instance is configured to store data for zero or more blobs in zero or more data stores and store metadata for all blobs stored at any local or global instance. The system selects one global instance to run a replication module that replicates blobs between instances according to blob policies. Some systems also include dynamic replication based on user needs.

Abstract translation: 分布式存储系统有多个实例。 存在多个本地实例，并且至少一些本地实例位于物理上不同的地理位置。 每个本地实例被配置为在具有多个不同数据存储类型的多个数据存储器中存储用于非空的一组Blob的数据。 此外，每个本地实例存储与数据存储不同的元数据存储中的相应组的组的元数据。 还有多个全局实例。 每个全局实例被配置为在零个或多个数据存储中存储零个或多个blob的数据，并存储在任何本地或全局实例中存储的所有blob的元数据。 系统选择一个全局实例来运行复制模块，该复制模块根据blob策略在实例之间复制Blob。 一些系统还包括基于用户需求的动态复制。

公开(公告)号：US20140304240A1

公开(公告)日：2014-10-09

申请号：US14293966

申请日：2014-06-02

Applicant: Google Inc.

Inventor： Yonatan Zunger ,  Alexandre Drobychev ,  Alexander Kesselman ,  Rebekah C. Vickrey ,  Frank C. Dachille ,  George Datuashvili

IPC: G06F17/30

CPC classification number: G06F17/30215 ,  G06F11/2094 ,  G06F11/3457 ,  H04L67/1095 ,  H04L67/1097

Abstract: A method allocates object replicas in a distributed storage system. The method identifies a plurality of objects in the distributed storage system. Each object has an associated storage policy that specifies a target number of object replicas stored at distinct instances of the distributed storage system. The method identifies an object of the plurality of objects whose number of object replicas exceeds the target number of object replicas specified by the storage policy associated with the object. The method selects a first replica of the object for removal based on last access times for replicas of the object, and transmits a request to a first instance of the distributed storage system that stores the first replica. The request instructs the first instance to remove the first replica of the object.

Abstract translation: 一种方法在分布式存储系统中分配对象副本。 该方法识别分布式存储系统中的多个对象。 每个对象具有关联的存储策略，其指定存储在分布式存储系统的不同实例处的对象副本的目标数量。 该方法识别多个对象的对象，其对象副本的数量超过与对象相关联的存储策略指定的对象副本的目标数量。 该方法基于对象的副本的最后访问时间选择要删除的对象的第一副本，并将请求发送到存储第一副本的分布式存储系统的第一实例。 请求指示第一个实例删除对象的第一个副本。