In some embodiments, a method determines when a packet is fragmented into multiple fragmented packets in a flow between a first workload and a second workload. The method switches from generating an outer source port in the outer header using layer 4 information from the inner header to using layer 3 information from the inner header. A fragmented packet is encapsulated with the outer header that includes an outer source port value that is generated using the layer 3 information. The method initiates a process to determine when to switch back to using layer 4 information from the inner header to generate the outer source port. When it is determined to switch back to using layer 4 information, the method switches back to using layer 4 information from the inner header to generate the source port in the outer header of a packet from the first workload.