A method for memory access includes maintaining in a host memory, under control of a host operating system running on a central processing unit (CPU), respective address translation tables for multiple processes executed by the CPU. Upon receiving, in a peripheral device, a work item that is associated with a given process, having a respective address translation table in the host memory, and specifies a virtual memory address, the peripheral device translates the virtual memory address into a physical memory address by accessing the respective address translation table of the given process in the host memory. The work item is executed in the peripheral device by accessing data at the physical memory address in the host memory.