Described herein are systems, methods, and software to manage replay windows in multipath connections between gateways. In one implementation, a first gateway may receive a packet directed toward a second gateway and identify a path from a plurality of paths to the second gateway. Once identified, the first gateway may increment a sequence number associated with the path and encapsulate the packet with a unique identifier for the path in the header with the incremented sequence number. The first gateway the communicates the encapsulated packet to the second gateway.