A personal audio device, such as a wireless telephone, includes an adaptive noise canceling (ANC) circuit that adaptively generates an anti-noise signal from a reference microphone signal and injects the anti-noise signal into the speaker or other transducer output to cause cancellation of ambient audio sounds. An error microphone is also provided proximate to the speaker to provide an error signal indicative of the effectiveness of the noise cancellation. A secondary path estimating adaptive filter is used to estimate the electro-acoustical path from the noise canceling circuit through the transducer so that source audio can be removed from the error signal. Adaptation of adaptive filters is sequenced so that update of their coefficients does not cause instability or error in the update. A level of the source audio with respect to the ambient audio can be determined to determine whether the system may generate erroneous anti-noise and/or become unstable.