公开(公告)号：US09619160B2

公开(公告)日：2017-04-11

申请号：US14839667

申请日：2015-08-28

Applicant: NetApp, Inc.

Inventor： Kayuri H. Patel ,  Hari Shankar

IPC: G06G3/06 ,  G06F3/06 ,  G06F12/02 ,  G06F11/14

CPC classification number: G06F3/0619 ,  G06F3/065 ,  G06F3/0659 ,  G06F3/0688 ,  G06F11/1441 ,  G06F11/1471 ,  G06F12/0246 ,  G06F2211/1038

Abstract: In one embodiment, a node coupled to a plurality of storage devices executes a storage input/output (I/O) stack having a plurality of layers including a persistence layer. A portion of non-volatile random access memory (NVRAM) is configured as one or more logs. The persistence layer cooperates with the NVRAM to employ the log to record write requests received from a host and to acknowledge successful receipt of the write requests to the host. The log has a set of entries, each entry including (i) write data of a write request and (ii) a previous offset referencing a previous entry of the log. After a power loss, the acknowledged write requests are recovered by replay of the log in reverse sequential order using the previous record offset in each entry to traverse the log.

公开(公告)号：US20150370498A1

公开(公告)日：2015-12-24

申请号：US14839667

申请日：2015-08-28

Applicant: NetApp, Inc.

Inventor： Kayuri H. Patel ,  Hari Shankar

IPC: G06F3/06

CPC classification number: G06F3/0619 ,  G06F3/065 ,  G06F3/0659 ,  G06F3/0688 ,  G06F11/1441 ,  G06F11/1471 ,  G06F12/0246 ,  G06F2211/1038

Abstract: In one embodiment, a node coupled to a plurality of storage devices executes a storage input/output (I/O) stack having a plurality of layers including a persistence layer. A portion of non-volatile random access memory (NVRAM) is configured as one or more logs. The persistence layer cooperates with the NVRAM to employ the log to record write requests received from a host and to acknowledge successful receipt of the write requests to the host. The log has a set of entries, each entry including (i) write data of a write request and (ii) a previous offset referencing a previous entry of the log. After a power loss, the acknowledged write requests are recovered by replay of the log in reverse sequential order using the previous record offset in each entry to traverse the log.

Abstract translation: 在一个实施例中，耦合到多个存储设备的节点执行具有包括持续层的多个层的存储输入/输出（I / O）堆栈。 非易失性随机存取存储器（NVRAM）的一部分被配置为一个或多个日志。 持久层与NVRAM配合使用日志来记录从主机接收到的写入请求，并确认成功接收写入请求给主机。 日志具有一组条目，每个条目包括（i）写入写入请求的数据，以及（ii）参考日志的先前条目的先前偏移量。 断电后，通过使用每个条目中的前一个记录偏移遍历日志，以相反的顺序重播日志来恢复确认的写入请求。

公开(公告)号：US20150193156A1

公开(公告)日：2015-07-09

申请号：US14151443

申请日：2014-01-09

Applicant: NetApp, Inc.

Inventor： Kayuri H. Patel ,  Hari Shankar

IPC: G06F3/06

CPC classification number: G06F3/0619 ,  G06F3/065 ,  G06F3/0659 ,  G06F3/0688 ,  G06F11/1441 ,  G06F11/1471 ,  G06F12/0246 ,  G06F2211/1038

Abstract: In one embodiment, a node coupled to a plurality of solid state drives (SSDs) executes a storage input/output (I/O) stack having a plurality of layers. Write data associated with one or more write requests to the SSDs is stored in a volatile log. The write data is organized into one or more extents that are copied to the SSDs. The volatile log has a front-end and a set of records with metadata. The metadata includes a head offset referencing an initial record and a tail offset referencing a final record. A portion of the one or more write requests including the write data is copied to a non-volatile log maintained in a non-volatile random access memory (NVRAM). The front-end and the set of records from the volatile log are copied, but the head offset and the tail offset are not, to reduce an amount of metadata copied to the NVRAM.

Abstract translation: 在一个实施例中，耦合到多个固态驱动器（SSD）的节点执行具有多个层的存储输入/输出（I / O）堆栈。 将与一个或多个写请求相关联的数据写入SSD被存储在易失性日志中。 写入数据被组织成一个或多个被复制到SSD的盘区。 易失性日志具有前端和一组具有元数据的记录。 元数据包括引用初始记录的头部偏移和引用最终记录的尾部偏移。 包括写入数据的一个或多个写请求的一部分被复制到维护在非易失性随机存取存储器（NVRAM）中的非易失性日志。 来自易失性日志的前端和一组记录被复制，但是头偏移和尾部偏移量不是，以减少复制到NVRAM的元数据量。

公开(公告)号：US09537710B2

公开(公告)日：2017-01-03

申请号：US14134063

申请日：2013-12-19

Applicant: NetApp Inc.

Inventor： Hari Shankar ,  Huadong Liu ,  Hua Li

IPC: G06F11/00 ,  H04L12/24 ,  G06F11/20

CPC classification number: H04L41/0654 ,  G06F11/2028 ,  G06F11/2038 ,  G06F11/2046 ,  G06F11/2097

Abstract: A novel RDMA connection failover technique that minimizes disruption to upper subsystem modules (executed on a computer node), which create requests for data transfer. A new failover virtual layer performs failover of an RDMA connection in error so that the upper subsystem that created a request does not have knowledge of an error (which is recoverable in software and hardware), or of a failure on the RDMA connection due to the error. Since the upper subsystem does not have knowledge of a failure on the RDMA connection or of a performed failover of the RDMA connection, the upper subsystem continues providing requests to the failover virtual layer without interruption, thereby minimizing downtime of the data transfer activity.

Abstract translation: 一种新颖的RDMA连接故障切换技术，可最大限度地减少上层子系统模块（在计算机节点上执行）的干扰，从而创建数据传输请求。 新的故障转移虚拟层错误地执行RDMA连接的故障转移，以便创建请求的上级子系统不知道错误（可在软件和硬件中恢复）或RDMA连接发生故障，因为 错误。 由于上级子系统不知道RDMA连接上的故障或RDMA连接的已执行故障转移，上级子系统会继续向故障转移虚拟层提供请求，而不会中断，从而最大限度地减少数据传输活动的停机时间。

公开(公告)号：US09323701B2

公开(公告)日：2016-04-26

申请号：US14446046

申请日：2014-07-29

Applicant: NetApp, Inc.

Inventor： Hari Shankar

IPC: G06F3/00 ,  G06F13/28 ,  G06F13/00 ,  G06F13/10 ,  G06F13/38

CPC classification number: G06F13/28 ,  G06F13/10 ,  G06F13/385 ,  G06F2221/2123

Abstract: A technique synchronizes de-registration of registered memory and incoming input/output (I/O) data received from an I/O device for storage in a memory of a computer system. Registration and de-registration of the memory with an I/O memory management unit (IOMMU) are illustratively performed by an I/O device driver of the computer system in anticipation of (or in response to) an I/O request to store the incoming I/O data in buffers of the memory. The synchronization technique ensures that storage of the I/O data in the buffers and de-registration of the buffers occur in a coordinated, reliable manner to obviate data corruption or other error conditions that may manifest in response to a race condition between such data storage and memory de-registration. Notably, I/O data which may be in-flight (i.e., inbound) from a sender to the I/O device may be received without error even when active buffers are deregistered. That is, the technique avoids handshaking with the sender before de-registering the active buffers.

Abstract translation: 一种技术将注册的存储器和从I / O设备接收的输入/输出（I / O）数据的注册同步，以存储在计算机系统的存储器中。 使用I / O存储器管理单元（IOMMU）的存储器的注册和取消注册在示例性地由计算机系统的I / O设备驱动程序执行，预期（或响应）I / O请求以存储 存储器缓冲区中的传入I / O数据。 同步技术确保缓冲器中的I / O数据的存储和缓冲器的取消注册以协调的，可靠的方式发生，从而避免数据损坏或其他错误条件，这些错误条件可能响应于这种数据存储之间的竞争条件 和记忆取消注册。 值得注意的是，即使当主动缓冲器被注销时，也可以毫无错误地接收从发送器到I / O设备的飞行中（即入站）的I / O数据。 也就是说，该技术在取消注册活动缓冲区之前避免与发送者握手。

公开(公告)号：US09152330B2

公开(公告)日：2015-10-06

申请号：US14151443

申请日：2014-01-09

Applicant: NetApp, Inc.

Inventor： Kayuri H. Patel ,  Hari Shankar

IPC: G06F12/16 ,  G06F3/06 ,  G06F12/02

CPC classification number: G06F3/0619 ,  G06F3/065 ,  G06F3/0659 ,  G06F3/0688 ,  G06F11/1441 ,  G06F11/1471 ,  G06F12/0246 ,  G06F2211/1038

Abstract: In one embodiment, a node coupled to a plurality of solid state drives (SSDs) executes a storage input/output (I/O) stack having a plurality of layers. Write data associated with one or more write requests to the SSDs is stored in a volatile log. The write data is organized into one or more extents that are copied to the SSDs. The volatile log has a front-end and a set of records with metadata. The metadata includes a head offset referencing an initial record and a tail offset referencing a final record. A portion of the one or more write requests including the write data is copied to a non-volatile log maintained in a non-volatile random access memory (NVRAM). The front-end and the set of records from the volatile log are copied, but the head offset and the tail offset are not, to reduce an amount of metadata copied to the NVRAM.

Abstract translation: 在一个实施例中，耦合到多个固态驱动器（SSD）的节点执行具有多个层的存储输入/输出（I / O）堆栈。 将与一个或多个写请求相关联的数据写入SSD被存储在易失性日志中。 写入数据被组织成一个或多个被复制到SSD的盘区。 易失性日志具有前端和一组具有元数据的记录。 元数据包括引用初始记录的头部偏移和引用最终记录的尾部偏移。 包括写入数据的一个或多个写请求的一部分被复制到维护在非易失性随机存取存储器（NVRAM）中的非易失性日志。 来自易失性日志的前端和一组记录被复制，但是头偏移和尾部偏移量不是，以减少复制到NVRAM的元数据量。

公开(公告)号：US08806115B1

公开(公告)日：2014-08-12

申请号：US14162289

申请日：2014-01-23

Applicant: NetApp, Inc.

Inventor： Kayuri H. Patel ,  Hari Shankar

IPC: G06F12/00

CPC classification number: G06F3/0619 ,  G06F3/065 ,  G06F3/0659 ,  G06F3/0688 ,  G06F11/1441 ,  G06F11/1471 ,  G06F12/0246 ,  G06F2211/1038

Abstract: In one embodiment, a parallel (e.g., tiered) logging technique is provided to deliver low latency acknowledgements of input/output (I/O) requests, such as write requests, while avoiding loss of data. Write data may be stored (copied) as a log in a portion of a dynamic random access memory and a non-volatile random access memory (NVRAM). The NVRAM may be configured as, e.g., a persistent write-back cache of the node, while parameters of the request may be stored in another portion of the NVRAM configured as the log (NVLog). The write data may be organized into separate variable length blocks or extents and “written back” out-of-order from the write-back cache to storage devices, such as SSDs, e.g., organized into a data container (intended destination of the write request). The write data may be preserved in the NVlog until each extent is safely stored on SSD.

Abstract translation: 在一个实施例中，提供并行（例如，分层）记录技术以在避免数据丢失的同时传送诸如写入请求的输入/输出（I / O）请求的低等待时间确认。 写数据可以作为日志存储（复制）在动态随机存取存储器和非易失性随机存取存储器（NVRAM）的一部分中。 NVRAM可以被配置为例如节点的持久回写缓存，而请求的参数可以被存储在被配置为日志（NVLog）的NVRAM的另一部分中。 写数据可以被组织成单独的可变长度块或扩展区，并且从写回高速缓存“无回写”到存储设备（例如SSD），例如被组织成数据容器（写入的预期目的地 请求）。 写入数据可以保存在NVlog中，直到每个盘区被安全地存储在SSD上。

公开(公告)号：US20160034407A1

公开(公告)日：2016-02-04

申请号：US14446046

申请日：2014-07-29

Applicant: NetApp, Inc.

Inventor： Hari Shankar

IPC: G06F13/28 ,  G06F13/10

CPC classification number: G06F13/28 ,  G06F13/10 ,  G06F13/385 ,  G06F2221/2123

Abstract: A technique synchronizes de-registration of registered memory and incoming input/output (I/O) data received from an I/0 device for storage in a memory of a computer system. Registration and de-registration of the memory with an I/O memory management unit (IOMMU) are illustratively performed by an I/O device driver of the computer system in anticipation of (or in response to) an I/O request to store the incoming I/O data in buffers of the memory. The synchronization technique ensures that storage of the I/O data in the buffers and de-registration of the buffers occur in a coordinated, reliable manner to obviate data corruption or other error conditions that may manifest in response to a race condition between such data storage and memory de-registration. Notably, I/O data which may be in-flight (i.e., inbound) from a sender to the I/O device may be received without error even when active buffers are deregistered. That is, the technique avoids handshaking with the sender before de-registering the active buffers.

Abstract translation: 一种技术将注册的存储器的解除注册和从I / O设备接收的输入/输出（I / O）数据同步，以存储在计算机系统的存储器中。 使用I / O存储器管理单元（IOMMU）的存储器的注册和取消注册在示例性地由计算机系统的I / O设备驱动程序执行，预期（或响应）I / O请求以存储 存储器缓冲区中的传入I / O数据。 同步技术确保缓冲器中的I / O数据的存储和缓冲器的取消注册以协调的，可靠的方式发生，以避免数据损坏或其他错误条件，这些错误条件可以响应于这种数据存储之间的竞争条件 和记忆取消注册。 值得注意的是，即使当主动缓冲器被注销时，也可以毫无错误地接收从发送器到I / O设备的飞行中（即入站）的I / O数据。 也就是说，该技术在取消注册活动缓冲区之前避免与发送者握手。

公开(公告)号：US20140136880A1

公开(公告)日：2014-05-15

申请号：US14134063

申请日：2013-12-19

Applicant: NetApp Inc.

Inventor： Hari Shankar ,  Huadong Liu ,  Hua Li

IPC: H04L12/24

CPC classification number: H04L41/0654 ,  G06F11/2028 ,  G06F11/2038 ,  G06F11/2046 ,  G06F11/2097

Abstract: A novel RDMA connection failover technique that minimizes disruption to upper subsystem modules (executed on a computer node), which create requests for data transfer. A new failover virtual layer performs failover of an RDMA connection in error so that the upper subsystem that created a request does not have knowledge of an error (which is recoverable in software and hardware), or of a failure on the RDMA connection due to the error. Since the upper subsystem does not have knowledge of a failure on the RDMA connection or of a performed failover of the RDMA connection, the upper subsystem continues providing requests to the failover virtual layer without interruption, thereby minimizing downtime of the data transfer activity.

Abstract translation: 一种新颖的RDMA连接故障切换技术，可最大限度地减少上层子系统模块（在计算机节点上执行）的干扰，从而创建数据传输请求。 新的故障转移虚拟层错误地执行RDMA连接的故障转移，以便创建请求的上级子系统不知道错误（可在软件和硬件中恢复）或RDMA连接发生故障，因为 错误。 由于上级子系统不知道RDMA连接上的故障或RDMA连接的已执行故障转移，上级子系统会继续向故障转移虚拟层提供请求，而不会中断，从而最大限度地减少数据传输活动的停机时间。