A personal audio device, such as a wireless telephone, includes an adaptive noise canceling (ANC) circuit that adaptively generates an anti-noise signal from a reference microphone signal and injects the anti-noise signal into the speaker or other transducer output to cause cancellation of ambient audio sounds. An error microphone is also provided proximate the speaker to provide an error signal indicative of the effectiveness of the noise cancellation. A secondary path estimating adaptive filter is used to estimate the electro-acoustical path from the noise canceling circuit through the transducer so that source audio can be removed from the error signal. Noise is injected either continuously and inaudibly below the source audio, or in response to detection that the source audio is low in amplitude, so that the adaptation of the secondary path estimating adaptive filter can be maintained, irrespective of the presence and amplitude of the source audio.