A system and method of performing speech arbitration at a client device that includes a neural network speech arbitration application, wherein the neural network speech arbitration application is configured to implement a neural network speech arbitration process, and wherein the method includes: receiving speech signals at a client device; generating and/or obtaining a set of inputs to be used in a speech arbitration neural network process, wherein the speech arbitration neural network process uses a neural network model that is tailored to speech arbitration and that can be used to determine whether and/or to what extent speech recognition processing of the received speech signals should be carried out at the client device; and receiving a speech arbitration output that indicates whether and/or to what extent the speech recognition processing of the received speech signals is to be carried out at the client device or at the remote server.