Abstract:
A storage gateway serves as an interface between processes on a customer network and a service provider. The storage gateway is located on-premise with the customer processes. To customer processes, it appears that data is stored locally. However, the storage gateway interfaces with a remote storage service to store the data. For cached gateways, the primary data store is a remote data store, while frequently accessed data may be locally cached by the gateway. Reads may be satisfied from the local cache or from virtual data storage; writes are handled so as to appropriately update data blocks in the local cache and/or in virtual data storage. For shadowing gateways, the primary data store is the local data store; reads are passed through to a local data store, and writes are shadowed to virtual data storage as well as being sent to local data store.
Abstract:
Methods and apparatus for efficient recovery of cached volumes at storage gateways are disclosed. To recover, after an unplanned shutdown, a storage gateway appliance configured to cache chunks of a storage object, chunk metadata corresponding to a particular chunk is read into an in-memory metadata region from a first metadata location. Based on analysis of the chunk metadata, a validation requirement indication for the particular chunk is stored, and the chunk is designated as being accessible for client I/O requests. In response to receiving a subsequent I/O request targeted to the particular chunk, the chunk metadata is validated using a different metadata location prior to performing the requested I/O operation.
Abstract:
Methods, apparatus, and computer-accessible storage media for implementing a gateway to a remote service provider according to a security model. The gateway serves as an interface between processes on a customer network and the provider, for example to store customer data to a remote data store. The model may include an activation process initiated by the gateway to register with the provider and associate the gateway with a customer account; the gateway is provided with security credentials. The model may also include establishing secure connections to external processes, for example processes of the service provider. The gateway initiates connections; the external processes do not initiate connections. The model may also include the customer managing the gateway through the service provider. The model may also include encrypting communications between the gateway and the provider and the gateway including security credentials in communications to the provider.
Abstract:
Methods and apparatus for secure data modification using segmented hashing are disclosed. An intermediate device on a data path between a storage service and a client receives a modification request for a data segment of a data chunk of a storage object. The device generates a new chunk hash value for the chunk based on an original chunk hash value of the chunk, an original segment hash value of the segment to be modified, and a new segment hash value computed based on the modification. The device generates a modified version of the chunk based on the modification request, and uploads the modified version and the new chunk hash value to the storage service.
Abstract:
Methods and apparatus for supporting cached volumes at storage gateways are disclosed. A storage gateway appliance is configured to cache at least a portion of a storage object of a remote storage service at local storage devices. In response to a client's write request, directed to at least a portion of a data chunk of the storage object, the appliance stores a data modification indicated in the write request at a storage device, and asynchronously uploads the modification to the storage service. In response to a client's read request, directed to a different portion of the data chunk, the appliance downloads the requested data from the storage service to the storage device, and provides the requested data to the client.
Abstract:
Methods and apparatus for WAN-optimized cache population at a storage gateway appliance are disclosed. A read request is received at the appliance from a client. The request is directed to a particular data block of a data chunk of a storage object of a remote storage service. A method comprises, in response to the read request, initiating a first data transfer of the data chunk from the remote storage service to an intermediate device. The method includes determining, based on one or more criteria, contents of one or more other data transfers including a second data transfer from the intermediate device to the appliance, initiating the second data transfer to transmit the requested data block to the appliance, and providing the particular data block to the client from the appliance.
Abstract:
A method and apparatus for staged execution pipelining and allocating resource to staged execution pipelines are provided. One or more execution pipelines are established, where each of the one or more execution pipelines includes one or more execution stages. Data is provided to the one or more execution pipelines for processing and resources are allocated to the execution pipeline.
Abstract:
A storage gateway serves as an interface between processes on a customer network and a service provider. The storage gateway is located on-premise with the customer processes. To customer processes, it appears that data is stored locally. However, the storage gateway interfaces with a remote storage service to store the data. For cached gateways, the primary data store is a remote data store, while frequently accessed data may be locally cached by the gateway. Reads may be satisfied from the local cache or from virtual data storage; writes are handled so as to appropriately update data blocks in the local cache and/or in virtual data storage. For shadowing gateways, the primary data store is the local data store; reads are passed through to a local data store, and writes are shadowed to virtual data storage as well as being sent to local data store.
Abstract:
A storage gateway serves as an interface between processes on a customer network and a service provider. The storage gateway is located on-premise with the customer processes. To customer processes, it appears that data is stored locally. However, the storage gateway interfaces with a remote storage service to store the data. For cached gateways, the primary data store is a remote data store, while frequently accessed data may be locally cached by the gateway. Reads may be satisfied from the local cache or from virtual data storage; writes are handled so as to appropriately update data blocks in the local cache and/or in virtual data storage. For shadowing gateways, the primary data store is the local data store; reads are passed through to a local data store, and writes are shadowed to virtual data storage as well as being sent to local data store.
Abstract:
Methods, apparatus, and computer-accessible storage media for shadowing data stored on a local store to a remote store provided by a service provider. A gateway may be configured as a shadowing gateway on a customer network in response to receiving configuration information. The shadowing gateway may receive reads and writes to the local store. The gateway passes the requests to the local store, and also uploads write data indicated by the writes to the service provider to update a snapshot of the local store maintained by the service provider on the remote store. The write data may be buffered to a write log for uploading, and may be uploaded as blocks according to a block storage format used by the service provider. The shadowing process may be transparent to processes on the customer network. The shadowed data may be used to recover data on the local store.