A personal audio device, such as a wireless telephone, includes noise canceling circuit that adaptively generates an anti-noise signal from a reference microphone signal and injects the anti-noise signal into the speaker or other transducer output to cause cancellation of ambient audio sounds. An error microphone may also be provided proximate the speaker to measure the output of the transducer in order to control the adaptation of the anti-noise signal and to estimate an electro-acoustical path from the noise canceling circuit through the transducer. A processing circuit that performs the adaptive noise canceling (ANC) function also detects frequency-dependent characteristics in and/or direction of the ambient sounds and alters adaptation of the noise canceling circuit in response to the detection.