A fuel pump is driven by a variable, constant current, control circuit. A pressure control valve disposed between the fuel pump and a fuel rail includes a downstream pressure control valve and a bypass valve bypassing the downstream pressure control valve. The downstream pressure control valve sets the fuel pressure in the fuel rail to a predetermined value by opening or closing a passage from a fuel pipe to the fuel rail. If the fuel pump discharge pressure is sufficiently increased, the bypass valve is opened when the upstream fuel pressure is larger than the downstream fuel pressure by at least a predetermined differential pressure. In this way, fuel is supplied to raise the fuel rail fuel pressure. Therefore, it is possible during other times to reduce the consumed electric power of the fuel pump and thus prolong its life.