In one embodiment, an apparatus includes: an access control circuit to receive a memory transaction directed to a storage, the memory transaction having a requester ID and a key ID; a first memory to store an access control table, the access control table having a plurality of entries each to store a requester ID and at least one key ID; and a cryptographic circuit coupled to the access control circuit, the cryptographic circuit to perform a cryptographic operation on data associated with the memory transaction based at least in part on the key ID. The apparatus may be implemented as an inline engine coupled between the storage and an accelerator, the inline engine to provide decrypted data to the accelerator, the storage to store encrypted data. Other embodiments are described and claimed.