One embodiment of the present invention provides an apparatus that reduces voltage noise for an integrated circuit (IC) device. This apparatus includes an interposer, which is configured to be sandwiched between the IC device and a circuit board. This interposer has a bottom surface, which is configured to receive electrical connections for power, ground and other signals from the circuit board, and a top surface, which is configured to provide electrical connections for power, ground and the other signals to the IC device. A plurality of bypass capacitors are integrated into the interposer and are coupled between the power and ground connections for the IC device, so that the plurality of bypass capacitors reduce voltage noise between the power and ground connections for the IC device.