Abstract:
An append-only data storage system is described that stores sets of data blocks in extents that are located in storage devices. During operation of the system, upon receiving a request to copy an extent from a source storage device to a destination storage device, the system creates a scratch extent on the destination storage device, and associates the scratch extent with a private identifier, whereby the scratch extent can only be accessed through the private identifier. The system uses the private identifier to perform a copying operation that copies the extent from the source storage device to the scratch extent on the destination storage device. After the copying operation is complete and the scratch extent is closed, the system associates the scratch extent with a public identifier, whereby the copy of the extent on the destination storage device becomes publically accessible to other entities in the data storage system.
Abstract:
The disclosed embodiments relate to the design of an append-only data storage system that stores sets of data blocks in extents that are located in storage devices in the system. During operation of the system, when an extent becomes full, the system changing the extent from an open state, wherein data can be appended to the extent, to a closed state, wherein data cannot be appended to the extent. Changing the extent from the open state to the closed state includes performing the following operations at one or more storage devices that contain copies of the extent: constructing an index to facilitate accessing data blocks in a copy of the extent contained in the storage device; and appending the index to the copy of the extent in non-volatile storage in the storage device.
Abstract:
A data storage system includes multiple zones that comprise separate geographic storage locations and store replicated copies of data items. Upon receiving a delete operation at a local zone at a time td, if a copy of the first data item exists in the local zone, the system computes a maximum last update time tmlu=td−tmin, wherein tmin is a minimum lifetime for a data item. Next, the system determines, from a local index, a time tlu that the first data item was last updated. If tlu
Abstract:
The disclosed embodiments relate to a system that uses colocation hints to facilitate storing data blocks in a distributed data storage system, which includes a plurality of data centers. During operation, the system receives a write request from a client to write a data block to the distributed data storage system, wherein the write request includes a colocation hint that identifies a colocation group associated with the data block. In response to the write request, the system uses the colocation hint to identify one or more data centers associated with the colocation group. Next, the system writes copies of the data block to the one or more identified data centers. In this way, the system situates copies of data blocks associated with the colocation group in the same data centers when possible.
Abstract:
A data storage system includes multiple storage zones that store replicated copies of data items. Upon receiving a delete operation at a particular time to delete a data item, the system computes a maximum last update time based on the particular time and a minimum data item lifetime. Based on determining that a copy of the data item is stored in the data storage zone, the system then determines a last update time of the data item. In addition, based on determining that the last update time is less than the first maximum last update time, the system deletes the copy of the data item from the data storage zone.
Abstract:
A system that stores sets of data blocks in extents located in storage devices is described. During operation, a receiving device receives, through an RPC framework, a first call asking to transfer an extent from a sending device to the receiving device. In response, the receiving device opens a port for a data connection that operates outside the RPC framework. The receiving device makes a second call, to the sending device through the RPC framework, asking to stream the extent to the port. The receiving device subsequently receives the extent from the sending device through the port and computes a checksum for the extent. The receiving device also receives a return from the second call, including a checksum for the extent computed by the sending device. If the computed checksum matches the received checksum, the receiving device returns the first call to indicate the transfer operation completed successfully.
Abstract:
The disclosed embodiments relate to the design of an append-only data storage system that stores sets of data blocks in extents that are located in storage devices in the system. During operation of the system, when an extent is in an open state, the system allows data blocks to be appended to the extent, and disallows operations to be performed on the extent that are incompatible with data being concurrently appended to the extent. When the extent becomes full, the system changes the extent from the open state to a closed state. Then, while the extent is in the closed state, the system disallows data blocks to be appended to the extent, and allows operations to be performed on the extent that are incompatible with data being concurrently appended to the extent.
Abstract:
A data storage system includes multiple storage zones that store replicated copies of data items. Upon receiving a delete operation at a particular time to delete a data item, the system computes a maximum last update time based on the particular time and a minimum data item lifetime. Based on determining that a copy of the data item is stored in the data storage zone, the system then determines a last update time of the data item. In addition, based on determining that the last update time is less than the first maximum last update time, the system deletes the copy of the data item from the data storage zone.
Abstract:
A data storage system includes multiple zones that comprise separate geographic storage locations and store replicated copies of data items. Upon receiving a delete operation at a local zone at a time td, if a copy of the first data item exists in the local zone, the system computes a maximum last update time tmlu=td−tmin, wherein tmin is a minimum lifetime for a data item. Next, the system determines, from a local index, a time tlu that the first data item was last updated. If tlu
Abstract:
A data storage system stores sets of data blocks in extents located on storage devices. During operation, the system performs an erasure-coding operation by obtaining a set of source extents, wherein each source extent is stored on a different machine in the data storage system. The system also selects a set of destination machines for storing destination extents, wherein each destination extent is stored on a different destination machine. Next, the system performs the erasure-coding operation by retrieving data from the set of source extents, performing the erasure-coding operation on the retrieved data to produce erasure-coded data, and then writing the erasure-coded data to the set of destination extents on the set of destination machines. Finally, after the erasure-coding operation is complete, the system commits results of the erasure-coding operation to enable the set of destination extents to be accessed in place of the set of source extents.