A speech-based system includes a local device in a user premises and a network-based control service that directs the local device to perform actions for a user. The control service may specify a first action that is to be performed upon detection by the local device of a stimulus. In some cases, performing the first action may rely on the availability of network communications with the control service or with another service. In these cases, the control service also specifies a second, fallback action that does not rely upon network communications. Upon detecting the stimulus, the local device performs the first action if network communications are available. If network communications are not available, the local device performs the second, fallback action.