A speech system may be configured to operate in conjunction with a stationary base device and a handheld remote device to receive voice commands from a user. Voice commands may be directed either to the base device or to the handheld device. When performing automatic speech recognition (ASR), natural language understanding (NLU), dialog management, text-to-speech (TTS) conversion, and other speech-related tasks, the system may utilize various models, including ASR models, NLU models, dialog models, and TTS models. Different models may be used depending on whether the user has chosen to speak into the base device or the handheld audio device. The different models may be designed to accommodate the different characteristics of audio and speech that are present in audio provided by the two different components and the different characteristics of the environmental situation of the user.