Abstract:
System and method for accessing a distributed storage system uses a storage-level access control process at a distributed file system that interfaces with the distributed storage system to determine whether a particular client has access to a particular first file system object using an identifier of the particular client and storage-level access control rules in response to a file system request from the particular client to access a second file system object in the particular first file system. The storage-level access control rules are defined for a plurality of clients and a plurality of first file system objects of the distributed storage system to allow the particular client access to the second file system object in the particular first file system object only if the particular client has been determined to have access to the particular first file system object according to the storage-level access control rules.
Abstract:
A transaction manager for handling operations on data in a storage system provides a system for executing transactions that uses a versioned tuple cache to achieve fast, abortable transactions using a redo-only log. The transaction manager updates an in-memory key-value store and also attaches a transaction identifier to the tuple as a minor key. Opportunistic locking can be accomplished due to the low cost of aborting transactions.
Abstract:
Systems and methods for allocating space in persistent storage are provided. A modified bitmap and a tree of bitmap summary pages are used to manage the free space of a large scale storage system. The bitmap is separated into fixed size pages and has bitmap summary entries to summarize the information in the bitmap. Bitmap summary pages can be further summarized into secondary summary pages. The tree data structure can continue to N levels until a topmost level has one bitmap summary page.
Abstract:
A distributed computing application is described that provides a highly elastic and multi-tenant platform for Hadoop applications and other workloads running in a virtualized environment. Deployments of a distributed computing application, such as Hadoop, may be executed concurrently with a distributed database application, such as HBase, using a shared instance of a distributed filesystem, or in other cases, multiple instances of the distributed filesystem. Computing resources allocated to region server nodes executing as VMs may be isolated from compute VMs of the distributed computing application, as well as from data nodes executing as VMs of the distributed filesystem.
Abstract:
A virtualized computing system for executing a distributed computing application, such as Hadoop, is discussed. The virtualized computing system stores data in a distributed filesystem, such as Hadoop Distributed File System, and processes data using a topology awareness that takes into account the virtualization layer of the virtualized computing system. The virtualized computing system employs locality-related policies, including replica placement policies, replica choosing policies, balancer policies, and task scheduling policies that take advantage of the awareness of the virtualization topology.
Abstract:
A file system uses a B-tree data structure to organize file data. The file system may maintain an index node (mode) representing a file and having entries that map to extents of the file. When the file system detects an index node, through updates, has exceeded a threshold number of extents, the file system converts the file to a copy-on-write (COW) B-tree data structure containing the entries representing the extents of the file. To clone the file, the file system uses copies of the index node and the root node of the COW B-tree data structure.
Abstract:
A sorted key-value store is implemented using a write-back cache maintained in memory, a B-tree data structured maintained in disk, and a logical and physical log for providing transactions. The logical log and write-back cache are used to answer client requests, while dirty blocks in the write-back cache are periodically flushed to disk using the physical log.
Abstract:
A distributed computing application is described that provides a highly elastic and multi-tenant platform for Hadoop applications and other workloads running in a virtualized environment. Production, test, and development deployments of a Hadoop application may be executed using multiple compute clusters and a shared instance of a distributed filesystem, or in other cases, multiple instances of the distributed filesystem. Data nodes executing as virtual machines (VMs) for test and development deployments can be linked clones of data nodes executing as VMs for a production deployment to reduce duplicated data and provide a shared storage space.