An apparatus and method are provided for handling incorrect branch direction predictions. The apparatus has processing circuitry for executing instructions, branch prediction circuitry for making branch direction predictions in respect of branch instructions, and fetch circuitry for fetching instructions from an instruction cache in dependence on the branch direction predictions and for forwarding the fetched instructions to the processing circuitry for execution. A cache location buffer stores cache location information for a given branch instruction for which accuracy of the branch direction predictions made by the branch prediction circuitry is below a determined threshold. The cache location information identifies where within the instruction cache one or more instructions are stored that will need to be executed in the event that a subsequent branch direction prediction made for the given branch instruction is incorrect. Control circuitry is responsive to such a subsequent branch direction prediction to obtain from the cache location buffer the cache location information and to provide the obtained cache location information to the fetch circuitry to enable the fetch circuitry to retrieve from the instruction cache the one or more instructions that are to be executed in the event that the processing circuitry does determine that the subsequent branch direction prediction was incorrect.