A speech-based system includes a local device in a user premises and a remote service that uses the local device to conduct speech dialogs with a user. The local device may also be directed to play audio such as music, audio books, etc. When designating audio for playing by the local device, the remote service may specify that the audio is either background audio or foreground audio. For background audio, the service indicates whether the background audio is mixable. For foreground audio, the service indicates an interrupt behavior. When the local device is playing background audio and receives foreground audio, the background audio is paused, attenuated, or not changed based on the indicated interrupt behavior of the foreground audio and whether the background audio has been designated as being mixable.