A personal audio device including earspeakers, includes an adaptive noise canceling (ANC) circuit that adaptively generates an anti-noise signal for each earspeaker from at least one microphone signal that measures the ambient audio, and the anti-noise signals are combined with source audio to provide outputs for the earspeakers. The anti-noise signals cause cancellation of ambient audio sounds at the respective earspeakers. A processing circuit uses the microphone signal(s) to generate the anti-noise signals, which can be generated by adaptive filters. The processing circuit controls adaptation of the adaptive filters such that when the processing circuit detects that either of the earspeakers are off-ear, a gain applied to the anti-noise signals is reduced.