An audio system has an ambient sound enhancement function, in which an against-the-ear audio device having a speaker converts a digitally processed version of an input audio signal into sound. The audio system also has an acoustic noise cancellation (ANC) function that may be combined in various ways with the sound enhancement function, and that may be responsive to voice activity detection. Other aspects are also described and claimed.