The disclosure relates to handover optimization in a voice communication session or other real-time multimedia session. For example, a wireless device associated with a parent user may detect a condition triggering a handover during a current voice communication session between the parent user and a target user, wherein the handover has a type associated with a switching delay. The wireless device may maintain statistics relating to an average mutual silence duration after talk spurts from the parent user and the target user based on one or more past voice communication sessions in which at least the parent user participated and trigger the handover when a next talk spurt from either the parent user or the target user ends where the average mutual silence duration exceeds the switching delay associated with the handover type.