A personal audio device includes a personal audio device housing; a transducer mounted on the housing for reproducing an audio signal including an anti-noise signal (anti-noise) for countering effects of ambient audio sounds in an acoustic output of the output transducer; a reference microphone mounted on the housing for providing a reference microphone signal (ref) indicative of the ambient audio sounds; and a processing circuit (30) within the housing. The processing circuit (30) adaptively generates the anti-noise signal (anti-noise) from the reference microphone signal (ref) such that the anti-noise signal (anti-noise) causes substantial cancellation of the ambient audio sounds. The processing circuit (30), in response to determining that an amplitude of acoustic leakage of the source audio into the reference microphone is substantial with respect to an amplitude of the ambient audio sounds, takes action to prevent improper generation of the anti-noise signal (anti-noise).