This is directed to controlling the output of audio based on the mode of an electronic device. In particular, this is directed to an electronic device having a sound mode, silent mode, and mute mode controlling the output of audio along different audio output routes. In the sound mode, all audio can be output, while only registered or authorized audio can be output in the silent mode. In the mute mode, no audio can be output. The sound and silent modes can be enabled using an interface of the device (e.g., a switch having two possible states), while the mute mode can be enabled using an accessory coupled to the device. To disengage the mute mode, a user can provide a corresponding instruction using the accessory, or providing an instruction on the device related to volume control. For example, a user can change the device volume using an on-device interface, or toggle a sound/silent switch to a sound mode.