In one embodiment, a processor includes circuitry to decode an instruction referencing an encoded data pointer that includes a set of plaintext linear address bits and a set of encrypted linear address bits. The processor also includes circuitry to perform a speculative lookup in a translation lookaside buffer (TLB) using the plaintext linear address bits to obtain physical address, buffer a set of architectural predictor state values based on the speculative TLB lookup, and speculatively execute the instruction using the physical address obtained from the speculative TLB lookup. The processor also includes circuitry to determine whether the speculative TLB lookup was correct and update a set of architectural predictor state values of the core using the buffered architectural predictor state values based on a determination that the speculative TLB lookup was correct.