Multi-modal applications are capable of interacting with a user in a voice mode, a visual mode, and/or other modes of communication. Mode swapping refers to the process of changing from one mode of communication (or one combination of modes) to another. Mode swapping may be initiated by a user, by an application, or by another entity. Mode swapping may be performed “adaptively,” in that an entity monitors the effectiveness of a communication mode and initiates a mode swap if communication is proceeding non-optimally in the current mode. Mode swapping may be performed by dropping a call to a telephone terminal in the currently-open mode and placing a call to that same device in another mode. In one embodiment, logic for mode swapping resides on a gateway proxy that mediates communication between an application and a telephone terminal.