公开(公告)号：US09418015B2

公开(公告)日：2016-08-16

申请号：US14695667

申请日：2015-04-24

Applicant: NetApp Inc.

Inventor： Rajesh Sundaram ,  Douglas Paul Doucette ,  David Grunwald ,  Jeffrey S. Kimmel ,  Ashish Prakash

IPC: G06F12/00 ,  G06F12/08 ,  G06F3/06

CPC classification number: G06F12/0866 ,  G06F3/0611 ,  G06F3/065 ,  G06F3/0656 ,  G06F3/068 ,  G06F3/0685 ,  G06F12/0802 ,  G06F2003/0691 ,  G06F2003/0692 ,  G06F2212/1024 ,  G06F2212/217 ,  G06F2212/222 ,  G06F2212/281

Abstract: Among other things, one or more techniques and/or systems are provided for storing data within a hybrid storage aggregate comprising a lower-latency storage tier and a higher-latency storage tier. In particular, frequently accessed data, randomly accessed data, and/or short lived data may be stored (e.g., read caching and/or write caching) within the lower-latency storage tier. Infrequently accessed data and/or sequentially accessed data may be stored within the higher-latency storage tier. Because the hybrid storage aggregate may comprise a single logical container derived from the higher-latency storage tier and the lower-latency storage tier, additional storage and/or file system functionality may be implemented across the storage tiers. For example, deduplication functionality, caching functionality, backup/restore functionality, and/or other functionality may be provided through a single file system (or other type of arrangement) and/or a cache map implemented within the hybrid storage aggregate.

公开(公告)号：US20160132396A1

公开(公告)日：2016-05-12

申请号：US15001401

申请日：2016-01-20

Applicant: NetApp, Inc.

Inventor： Jeffrey S. Kimmel ,  T. Byron Rakitzis

IPC: G06F11/14 ,  G06F17/30

CPC classification number: G06F11/1407 ,  G06F11/1417 ,  G06F11/1438 ,  G06F11/1464 ,  G06F11/1471 ,  G06F12/0246 ,  G06F16/128 ,  G06F16/2228 ,  G06F16/2255 ,  G06F2201/80 ,  G06F2201/84

Abstract: In one embodiment, an extent store layer of a storage input/output (I/O) stack executing on one or more nodes of a cluster manages efficient logging and checkpointing of metadata. The metadata managed by the extent store layer, i.e., the extent store metadata, resides in a memory (in-core) of each node and is illustratively organized as a key-value extent store embodied as one or more data structures, e.g., a set of hash tables. Changes to the set of hash tables are recorded as a continuous stream of changes to SSD embodied as an extent store layer log. A separate log stream structure (e.g., an in-core buffer) may be associated respectively with each hash table such that changed (i.e., dirtied) slots of the hash table are recorded as entries in the log stream structure. The hash tables are written to SSD using a fuzzy checkpointing technique.

Abstract translation: 在一个实施例中，在集群的一个或多个节点上执行的存储输入/输出（I / O）堆栈的盘区存储层管理元数据的有效日志记录和检查点。 由盘区存储层管理的元数据，即盘区存储元数据，驻留在每个节点的存储器（内核）中，并且被说明性地组织为体现为一个或多个数据结构的键值范围存储，例如， 一组哈希表。 对这组哈希表的更改记录为作为扩展存储层日志实现的SSD的连续变化流。 单独的日志流结构（例如，核内缓冲器）可以分别与每个散列表相关联，使得散列表的改变（即，脏的）时隙被记录在日志流结构中作为条目。 使用模糊检查点技术将哈希表写入SSD。

公开(公告)号：US09170751B2

公开(公告)日：2015-10-27

申请号：US14341541

申请日：2014-07-25

Applicant: NetApp, Inc.

Inventor： Jeffrey S. Kimmel

IPC: G06F12/00 ,  G06F3/06

CPC classification number: G06F3/065 ,  G06F3/0614 ,  G06F3/0619 ,  G06F3/0626 ,  G06F3/0655 ,  G06F3/0659 ,  G06F3/067 ,  G06F3/0679 ,  G06F3/0685 ,  G06F3/0689 ,  G06F12/00 ,  G06F2206/1014

Abstract: A storage server receives a write request from a client system including new data and a location to store the new data. The storage server transmits a copy instruction to a storage subsystem to relocate old data at the location and transmits a write instruction to the storage subsystem to overwrite the old data with the new data. The storage subsystem includes fast stable storage in which the copy instruction and the write instruction are stored. After receiving each instruction, the storage subsystem sends an acknowledgement to the storage server. When both instructions have been acknowledged, the storage server sends an acknowledgement to the client system. The storage subsystem performs the instructions asynchronously from the client system's write request.

Abstract translation: 存储服务器从包括新数据和存储新数据的位置的客户端系统接收写入请求。 存储服务器将复制指令发送到存储子系统以重新定位该位置处的旧数据，并向存储子系统发送写指令以用新数据覆盖旧数据。 存储子系统包括存储复制指令和写指令的快速稳定存储。 收到每条指令后，存储子系统向存储服务器发送确认。 当两个指令都被确认时，存储服务器向客户端系统发送确认。 存储子系统从客户端系统的写请求异步执行指令。

公开(公告)号：US09152335B2

公开(公告)日：2015-10-06

申请号：US14160216

申请日：2014-01-21

Applicant: NetApp, Inc.

Inventor： Rajesh Sundaram ,  Jeffrey S. Kimmel ,  Blake H. Lewis

IPC: G06F12/12 ,  G06F3/06

CPC classification number: G06F3/0641 ,  G06F3/0604 ,  G06F3/0608 ,  G06F3/0643 ,  G06F3/067 ,  G06F3/0688 ,  G06F2003/0697

Abstract: In one embodiment, a layered file system of a storage input/output (I/O) stack executes on one or more nodes of a cluster. The layered file system includes a flash-optimized, log-structured layer configured to provide sequential storage of data and metadata (i.e., a log-structured layout) on solid state drives (SSDs) of storage arrays in the cluster to reduce write amplification, while leveraging a data de-duplication feature of the storage I/O stack. An extent store layer of the file system performs and maintains mappings of the extent keys to SSD storage locations, while a volume layer of the file system performs and maintains mappings of the LUN offset ranges to the extent keys. Separation of the mapping functions between the volume and extent store layers enables different volumes with different offset ranges to reference a same extent key (and thus a same extent).

Abstract translation: 在一个实施例中，存储输入/输出（I / O）堆栈的分层文件系统在集群的一个或多个节点上执行。 分层文件系统包括闪存优化的日志结构化层，其被配置为在集群中的存储阵列的固态驱动器（SSD）上提供数据和元数据（即，日志结构化布局）的顺序存储以减少写入放大， 同时利用存储I / O堆栈的重复数据删除功能。 文件系统的扩展存储层执行并维护扩展密钥到SSD存储位置的映射，而文件系统的卷层执行并维护LUN偏移范围到扩展密钥的映射。 卷和卷存储层之间的映射函数的分离使得具有不同偏移范围的不同卷可以引用相同的扩展密钥（并因此相同的扩展）。

公开(公告)号：US20150193338A1

公开(公告)日：2015-07-09

申请号：US14150717

申请日：2014-01-08

Applicant: NetApp, Inc.

Inventor： Rajesh Sundaram ,  Stephen Daniel ,  Jeffrey S. Kimmel ,  Blake H. Lewis

IPC: G06F12/02 ,  G06F3/06

CPC classification number: G06F17/30138 ,  G06F3/0604 ,  G06F3/0608 ,  G06F3/0616 ,  G06F3/064 ,  G06F3/0643 ,  G06F3/0644 ,  G06F3/0647 ,  G06F3/0659 ,  G06F3/0679 ,  G06F3/0685 ,  G06F3/0688 ,  G06F12/0246 ,  G06F17/30185 ,  G06F17/30218 ,  G06F2003/0697 ,  G06F2212/2022 ,  G06F2212/7208

Abstract: In one embodiment, storage arrays of solid state drives (SSDs) coupled to a node are organized as redundant array of independent disks (RAID) groups. Each storage array includes one or more segments. Each segment has contiguous free space on the SSDs. Data and metadata is organized on the SSDs with a sequential log-structured layout, with the data organized as variable-length extents of one or more logical units (LUNs). Segment cleaning is performed to clean a selected segment by moving the extents of the selected segment that contain valid data to one or more different segments so as to free the selected segment. Additional extents are written as a sequence of contiguous range write operations to the entire free segment with temporal locality to reduce data relocation within the SSDs as a result of the write operations.

Abstract translation: 在一个实施例中，耦合到节点的固态驱动器（SSD）的存储阵列被组织为独立磁盘（RAID）组的冗余阵列。 每个存储阵列包括一个或多个段。 每个段在SSD上具有连续的可用空间。 数据和元数据在具有顺序日志结构布局的SSD上组织，数据组织为一个或多个逻辑单元（LUN）的可变长度范围。 执行段清洁以通过将包含有效数据的所选段的范围移动到一个或多个不同段来清理所选择的段，以便释放所选择的段。 作为连续范围写入操作的序列，附加范围被写入具有时间局部性的整个空闲段，以减少作为写入操作的结果的SSD内的数据重定位。

公开(公告)号：US20150193170A1

公开(公告)日：2015-07-09

申请号：US14150689

申请日：2014-01-08

Applicant: NetApp, Inc.

Inventor： Rajesh Sundaram ,  Jeffrey S. Kimmel ,  Blake H. Lewis

IPC: G06F3/06

CPC classification number: G06F3/0641 ,  G06F3/0604 ,  G06F3/0608 ,  G06F3/0643 ,  G06F3/067 ,  G06F3/0688 ,  G06F2003/0697

Abstract: In one embodiment, a layered file system includes a volume layer and an extent store layer configured to provide sequential log-structured layout of data and metadata on solid state drives (SSDs) of one or more storage arrays. The data is organized as variable-length extents of one or more logical units (LUNs). The metadata includes volume metadata mappings from offset ranges of a LUN to extent keys and extent metadata mappings of the extent keys to storage locations of the extents on the SSDs. The extent store layer maintaining the extent metadata mappings determines whether an extent is stored on a storage array, and, in response to determination that the extent is stored on the storage array, returns an extent key for the stored extent to the volume layer to enable global inline de-duplication that obviates writing a duplicate copy of the extent on the storage array.

Abstract translation: 在一个实施例中，分层文件系统包括卷层和盘区存储层，其被配置为在一个或多个存储阵列的固态驱动器（SSD）上提供数据和元数据的顺序日志结构布局。 数据被组织为一个或多个逻辑单元（LUN）的可变长度范围。 元数据包括从LUN的偏移范围到扩展密钥的卷元数据映射以及扩展密钥到SSD上的扩展区的存储位置的扩展元数据映射。 维护盘区元数据映射的盘区存储层确定盘区是否存储在存储阵列上，并且响应于确定该盘区存储在存储阵列上，将存储的盘区的盘区密钥返回到卷层以启用 全局内联重复数据删除，从而避免在存储阵列上写入扩展区的重复副本。

公开(公告)号：US20150193169A1

公开(公告)日：2015-07-09

申请号：US14160216

申请日：2014-01-21

Applicant: NetApp, Inc.

Inventor： Rajesh Sundaram ,  Jeffrey S. Kimmel ,  Blake H. Lewis

IPC: G06F3/06

CPC classification number: G06F3/0641 ,  G06F3/0604 ,  G06F3/0608 ,  G06F3/0643 ,  G06F3/067 ,  G06F3/0688 ,  G06F2003/0697

Abstract: In one embodiment, a layered file system of a storage input/output (I/O) stack executes on one or more nodes of a cluster. The layered file system includes a flash-optimized, log-structured layer configured to provide sequential storage of data and metadata (i.e., a log-structured layout) on solid state drives (SSDs) of storage arrays in the cluster to reduce write amplification, while leveraging a data de-duplication feature of the storage I/O stack. An extent store layer of the file system performs and maintains mappings of the extent keys to SSD storage locations, while a volume layer of the file system performs and maintains mappings of the LUN offset ranges to the extent keys. Separation of the mapping functions between the volume and extent store layers enables different volumes with different offset ranges to reference a same extent key (and thus a same extent).

Abstract translation: 在一个实施例中，存储输入/输出（I / O）堆栈的分层文件系统在集群的一个或多个节点上执行。 分层文件系统包括闪存优化的日志结构化层，其被配置为在集群中的存储阵列的固态驱动器（SSD）上提供对数据和元数据的顺序存储（即，日志结构化布局），以减少写入放大， 同时利用存储I / O堆栈的重复数据删除功能。 文件系统的扩展存储层执行并维护扩展密钥到SSD存储位置的映射，而文件系统的卷层执行并维护LUN偏移范围到扩展密钥的映射。 卷和卷存储层之间的映射函数的分离使得具有不同偏移范围的不同卷可以引用相同的扩展密钥（并因此相同的扩展）。

公开(公告)号：US09071622B2

公开(公告)日：2015-06-30

申请号：US14101078

申请日：2013-12-09

Applicant: NetApp, Inc.

Inventor： David Grunwald ,  Jeffrey S. Kimmel

IPC: G06F9/46 ,  H04L29/08 ,  G06F17/30 ,  G06F9/50 ,  G06F9/52

CPC classification number: H04L67/1087 ,  G06F9/5011 ,  G06F9/526 ,  G06F17/30224 ,  G06F2209/506

Abstract: A technique is described for improving throughput in a processing system, such as a network storage server. The technique provides multiple levels (e.g., a hierarchy) of parallelism of process execution within a single mutual exclusion domain, in a manner which allows certain operations on metadata to be parallelized as well as certain operations on user data. The specific parallelization scheme used in any given embodiment is based at least partly on the underlying metadata structures used by the processing system. Consequently, a high degree of parallelization possible, which improves the throughput of the processing system.

Abstract translation: 描述了一种用于提高诸如网络存储服务器的处理系统中的吞吐量的技术。 该技术以允许对元数据进行某些操作并行化以及对用户数据的某些操作的方式在单个互斥域内提供过程执行的并行性的多个级别（例如，层级）。 在任何给定实施例中使用的特定并行化方案至少部分地基于处理系统使用的底层元数据结构。 因此，可以进行高度并行化，这提高了处理系统的吞吐量。

公开(公告)号：US09043542B2

公开(公告)日：2015-05-26

申请号：US14087557

申请日：2013-11-22

Applicant: NetApp, Inc.

Inventor： Jeffrey S. Kimmel ,  Randy Pafford ,  Rajesh Sundaram

IPC: G06F12/00 ,  G06F12/12 ,  G06F12/02

CPC classification number: G06F12/12 ,  G06F12/0246 ,  G06F2212/222 ,  G06F2212/7211

Abstract: Described is a technique for managing the content of a nonvolatile solid-state memory data cache to improve cache performance while at the same time, and in a complementary manner, providing for automatic wear leveling. A modified circular first-in first-out (FIFO) log/algorithm is generally used to determine cache content replacement. The algorithm is used as the default mechanism for determining cache content to be replaced when the cache is full but is subject to modification in some instances. In particular, data are categorized according to different data classes prior to being written to the cache, based on usage. Once cached, data belonging to certain classes are treated differently than the circular FIFO replacement algorithm would dictate. Further, data belonging to each class are localized to designated regions within the cache.

Abstract translation: 描述了一种用于管理非易失性固态存储器数据高速缓存的内容以提高高速缓存性能的技术，同时以互补的方式提供自动磨损均衡。 修改的循环先进先出（FIFO）日志/算法通常用于确定高速缓存内容替换。 该算法用作在高速缓存已满的情况下确定要替换的缓存内容的默认机制，但在某些情况下可能会进行修改。 特别地，根据使用情况，在写入缓存之前，根据不同的数据类对数据进行分类。 一旦缓存，属于某些类的数据的处理方式与循环FIFO替换算法将会有所不同。 此外，属于每个类的数据被定位到高速缓存内的指定区域。

公开(公告)号：US20150095347A1

公开(公告)日：2015-04-02

申请号：US14160048

申请日：2014-01-21

Applicant: NetApp, Inc.

Inventor： Jeffrey S. Kimmel ,  Blake H. Lewis

IPC: G06F17/30

CPC classification number: G06F17/3033 ,  G06F3/0608 ,  G06F3/0641 ,  G06F3/067 ,  G06F17/30097 ,  G06F17/30949

Abstract: In one embodiment, an extent hashing technique is used to efficiently distribute data and associated metadata substantially evenly among nodes of a cluster. The data may be write data associated with a write request issued by a host and received at a node of the cluster. The write data may be organized into one or more extents. A hash function may be applied to the extent to generate a result which may be truncated or trimmed to generate a hash value. A hash space of the hash value may be divided into a plurality of buckets representative of the write data, i.e., the extents, and the associated metadata, i.e., extent metadata. A number of buckets may be assigned to each extent store instance of the nodes to distribute ownership of the buckets, along with their extents and extent metadata, across all of the extent store instances of the nodes.

Abstract translation: 在一个实施例中，使用扩展散列技术来有效地在簇的节点之间均匀地分布数据和相关联的元数据。 数据可以是与由主机发出并在集群的节点处接收的写入请求相关联的写入数据。 写入数据可以被组织成一个或多个范围。 可以将哈希函数应用于生成可能被截断或修剪以生成哈希值的结果的程度。 哈希值的散列空间可以被划分为代表写入数据的多个存储桶，即扩展区以及相关联的元数据，即扩展元数据。 可以将多个桶分配给节点的每个盘区存储实例，以在节点的所有范围存储实例之间分配桶的所有权以及它们的盘区和盘区元数据。