Embodiments of the invention describe systems, apparatuses and methods that enable sharing Remote Direct Memory Access (RDMA) device hardware between a host and a peripheral device including a CPU and memory complex (alternatively referred to herein as a processor add-in card). Embodiments of the invention utilize interconnect hardware such as Peripheral Component Interconnect express (PCIe) hardware for peer-to-peer data transfers between processor add-in cards and RDMA devices. A host system may include modules or logic to map memory and registers to and/or from the RDMA device, thereby enabling I/O to be performed directly to and from user-mode applications on the processor add-in card, concurrently with host system I/O operations.