Implementations disclosed herein relate to generating and/or utilizing, by a client device, a text-response map that is stored locally on the client device. The text-response map can include a plurality of mappings, where each of the mappings define a corresponding direct relationship between corresponding text and a corresponding response. Each of the mappings is defined in the text-response map based on the corresponding text being previously generated from previous audio data captured by the client device and based on the corresponding response being previously received from a remote system in response to transmitting, to the remote system, at least one of the previous audio data and the corresponding text.