Methods, and systems, including computer programs encoded on computer-readable storage mediums, including a method for performing semantic interpretation using gaze order. The method includes obtaining data identifying a sequence of gaze attention dwell positions, obtaining a semantic description of elements displayed on a visual display, obtaining a transcription of an utterance, correlating the gaze attention dwell positions with the semantic description of elements to generate a sequence of one or more of the elements, performing semantic interpretation of at least one term included in the transcription based at least on the sequence of the elements, and outputting a result of performing the semantic interpretation of the at least one term.