In one embodiment, a processor includes a memory hierarchy that stores encrypted data, tracking circuitry that tracks an execution context for instructions executed by the processor, and cryptographic computing circuitry to encrypt/decrypt data that is stored in the memory hierarchy. The cryptographic computing circuitry obtains context information from the tracking circuitry for a load instruction to be executed by the processor, where the context information indicates information about branch predictions made by a branch prediction unit of the processor, and decrypts the encrypted data using a key and the context information as a tweak input to the decryption.