Embodiments of the present invention recite a method and system for improving the fidelity of a dialog system. In one embodiment, a first input generated by a user of a first system operating in a first modality is accessed. In embodiments of the present invention, the first system also generates a first output corresponding to the first input. An second input from a second user, who is engaged in a conversation with the first user, is accessed by a second system. The second input is then utilized to modify the first output of the first system.