Hardware mechanisms are provided for performing hardware based access control of instructions to data. These hardware mechanisms associate an instruction access policy label with an instruction to be processed by a processor and associate an operand access policy label with data to be processed by the processor. The instruction access policy label is passed along with the instruction through one or more hardware functional units of the processor. The operand access policy label is passed along with the data through the one or more hardware functional units of the processor. One or more hardware implemented policy engines associated with the one or more hardware functional units of the processor are utilized to control access by the instruction to the data based on the instruction access policy label and the operand access policy label.