A system for controlling multiple devices using automatic speech recognition (ASR) even when the devices may not be capable of performing ASR themselves. A device such as a media player, appliance, or the like may be recognized by a network. The configured controls for the device (such as a remote control or other mechanism) are incorporated into a device control registry which catalogs device command controls. Individual ASR grammars are constructed for the devices so speech commands for those devices may be processed by an ASR device. The ASR device may then process those speech commands and convert them into the appropriate inputs for the controlled device. The inputs may then be sent to the controlled device, resulting in ASR control for non-ASR devices.