A gas turbine engine includes a main compressor section with a downstream most location. A turbine section has a high pressure turbine. A tap line is connected to tap air from a location upstream of the downstream most location in the main compressor section. The tapped air is connected to a heat exchanger and then to a cooling compressor. The cooling compressor compresses air downstream of the heat exchanger, and is connected to deliver air into the high pressure turbine. A bypass valve is positioned downstream of the main compressor section, and upstream of the heat exchanger. The bypass valve selectively delivers air directly to the cooling compressor without passing through the heat exchanger under certain conditions.