A gas turbine engine has a fan rotor delivering air into a bypass duct defined between an outer fan case and an outer interior housing. The fan rotor also delivers air into a compressor section, a combustor, a turbine section. A chamber is defined between the outer interior housing and an inner housing. The inner housing contains the compressor section, the combustor and the turbine section. A first conduit taps hot compressed air to be cooled and passes the air to at least one heat exchanger. The air is cooled in the heat exchanger and returned to a return conduit. The return conduit passes the cooled air to at least one of the turbine section and the compressor section. The heat exchanger has a core exhaust plane. The turbine section has at least a first and a downstream second rotor blade row, with the core exhaust plane located downstream of a center plane of the second blade row.