A video conference cross-links at least a portion of the clients for point to point communication while still using a video conference server. The video conference server continues to manage some aspects of the video conference for the cross-linked clients and may be configured to perform audio/video processing for some of the clients in a video conference. For example, the video conference server may perform audio/video processing and routing of streams for clients that do not have the capabilities to perform the processing directly. Clients that are able to process their audio/video needs may directly connect to other clients through cross-links while still using the video conference server for management.