A personal audio device, such as a wireless telephone, includes an adaptive noise canceling (ANC) circuit that generates an anti-noise signal from a reference microphone signal and injects the anti-noise signal into the speaker or other transducer output to cancel ambient audio sounds. A processing circuit implements one or more adaptive filters that control the generation of the anti-noise signal. At least one of the adaptive filters is partitioned into a first portion having a fixed frequency response and a second portion having a variable frequency response. The partitioned filter may be an adaptive filter that generates the anti-noise signal directly from the reference microphone signal. An error microphone may be provided to measure the ambient sounds and transducer output near the transducer, and a secondary path adaptive filter included to generate an error signal from the error microphone signal, which may be partitioned, alone or in combination.