A personal audio device including earspeakers, includes an adaptive noise canceling (ANC) circuit that adaptively generates an anti-noise signal for each earspeaker from at least one microphone signal that measures the ambient audio, and the anti-noise signals are combined with source audio to provide outputs for the earspeakers. The anti-noise signals cause cancellation of ambient audio sounds at the respective earspeakers. A processing circuit uses the microphone signal(s) to generate the anti-noise signals, which can be generated by adaptive filters. The processing circuit controls adaptation of the adaptive filters such that when an event requiring action on the adaptation of one of the adaptive filters is detected, action is taken on the other one of the adaptive filters. Another feature of the ANC system uses microphone signals provided at both of the earspeakers to perform processing on a voice microphone signal that receives speech of the user.