A method comprises establishing a video conference between at least two instances of a media application over a relay connection; in response to determining that the video conference comprises only two instances of the media application: receiving, from a first instance of the two instances of the media application, a plurality of candidate addresses for a peer to peer (P2P) connection with a second instance of the two instances of the media application; sending, to the second instance of the media application, the plurality of candidate addresses, wherein the P2P connection is established using a particular candidate address of the plurality of candidate addresses; after the P2P is established, maintaining the relay connection in an inactive state while media data for the video conference is sent over the P2P connection until a switch-over event occurs, wherein metadata is sent over the relay connection to keep the relay connection in the inactive state.