A memory manager on a programmable device manages memory allocated to accelerators on the programmable device and allocated to processes that access the programmable device. The memory manager can manage both memory on the programmable device as well as external memory coupled to the programmable device. The memory manager protects the memory from unauthorized access by enforcing protection for the memory, using keys, encryption or the like. The memory manger can allocate a partition of memory to an accelerator when an accelerator is deployed to a programmable device, then allocate subpartitions within the allocated partition for each process that accesses the accelerator. When an accelerator is cast out of the programmable device, the memory partition is scrubbed so it can be reclaimed and allocated to another accelerator. When a process terminates, the subpartitions corresponding to the process are scrubbed so they may be reclaimed and allocated to another process.