One embodiment of the present invention provides a system for predicting an address of an instruction following a branch instruction in a computer instruction stream. This system receives a current address specifying an address of a current instruction. It uses this current address (or possibly a preceding address) to generate a first select signal, which is used to select a first predicted address of an instruction following the current instruction in the computer instruction stream. At the same time the system generates a second select signal, which takes more time to generate than the first select signal but achieves a more accurate selection for a predicted address of the instruction following the current instruction. The system assumes that the first predicted address is correct and proceeds with a subsequent instruction fetch operation using the first predicted address. Next, the system compares the first select signal with the second select signal. If the first select signal is the same as the second select signal, the system allows the subsequent instruction fetch operation to proceed using the first predicted address. Otherwise, the system uses the second select signal to select a second predicted address, and delays the subsequent instruction fetch operation so that the instruction fetch operation can proceed using the second predicted address. This bi-level architecture allows branch prediction work efficiently even at the higher clock frequencies that arise as semiconductor technologies continue to improve.