A gas turbine engine has a compressor section including a lower pressure compressor and a higher pressure compressor, and a turbine section. A core engine housing surrounds the compressor section and the turbine section. An outer intermediate housing wall defines an internal chamber between the core housing and the outer intermediate housing. A fan rotor and a fan casing surround the fan rotor to define a bypass duct between the fan case and the outer intermediate housing. A heat exchanger is mounted in the internal chamber and receives high pressure air for cooling the high pressure air and delivering the high pressure air into the core engine housing to be utilized as cooling air for a component. Air from the lower pressure compressor is utilized to cool the higher pressure air in the heat exchanger.