Use Case and Requirements for Flow Control and Congestion Control Coordination in WAN-based AI DC Interconnect

Driven by the need for larger model scales, higher compute density, and improved resource utilization, AI services are moving from single-site data center (DC) toward a multi-site distributed architecture. This requires to provide high-throughput, lossless, and deterministic network connectivity for AI workloads that span Data Center Networks (DCNs) over the Wide Area Network (WAN). Any packet loss or tail latency directly translates into stalled training pipelines, degraded job completion times (JCT), and wasted GPU cycles. A technical challenge arises from the fundamental difference between DCN and DCI scenarios is in congestion control and flow control. Within a single data center, the short physical distance keeps the Round-Trip Time (RTT) low, typically in the order of tens of microseconds, allowing the control loop to react quickly and maintain a stable, lossless fabric through mechanisms such as Explicit Congestion Notification (ECN), Congestion Notification Packet (CNP) and Priority Flow Control (PFC). In the distributed DCI scenario, the physical distance between DCs stretches the RTT to milliseconds or even tens of milliseconds. This significantly slows down the control loop, making it difficult to promptly detect and react to congestion in the network. As a result, queue buildup and packet loss become far more likely, and the tight lossless guarantee achievable in DCN cannot be easily extended to the WAN domain. This motivates the need for coordinated congestion control and flow control across the WAN and the data center. describes the problem and limitations of existing approaches and documents the need for a fast notification mechanism. provides the overall architecture, the functional role and information carried in fast network notifications. Rather than treating DC and WAN as independent network domains, AI DCI requires the congestion control and flow control mechanisms in both domains to work in an coordinated manner to preserve end-to-end lossless, high bandwidth utilization characteristics, so as to meet the performance demands of distributed AI workloads. This document describes the use case of coordinated congestion control and flow control in WAN based AI DCI to minimize the packet loss in both DCN and WAN networks. The requirements on a fast network notification mechanism in this scenraio is also described.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

The following terminologies are used in this document: PFC: Priority-based Flow Control ECN: Explicit Congestion Notification CNP: Congestion Notification Packet RoCEv2: RDMA over Converged Ethernet Version 2

Figure 1 shows a typical topology of distributed DC interconnect, in which two or more DCs are interconnected via the WAN network. The WAN network may be used only for AI DCI traffic, or it may also be used to carry other AI or non-AI services. Congestion may happen either within the DCs or in the WAN. The following of this section describes different congestion scenarios, and the coordination mechanisms in flow control and congestion control.

Assume AI traffic is sent from DC-A (the ingress DC) to DC-B (the egress DC) taking the path R1-R3-R5 in WAN network. If a traffic burst or surge causes congestion on link R3-R5 (as shown in Figure 1), node R3 has the following options for handling congestion: Option 1: Receiver-Based Congestion Notification (Standard ECN) R3 marks the ECN field in the data packet as "Congestion Experienced (CE)" and forwards it to the receiver in the egress DC (DC-B). Upon detecting the CE mark, the receiver in DC-B generates a congestion notification message and sends it back to the traffic source in the ingress DC (DC-A), instructing the source to reduce its sending rate for the affected flows. This is the standard congestion notification mechanism used in DCN. However, in a WAN-based DC Interconnect scenario, the control loop spans the full WAN RTT, which is typically in the millisecond range. This latency may prevent timely rate reduction, leading R3 to drop packets due to buffer overflow.

+----------+ Congestion +----------+ | Data | +----+ +----+ +----+ | Data | | center A |---| R1 |---| R3 |-=-| R5 |---| center B | +----------+ +----+ +----+ +----+ +----------+ Packet with ECN Set |--------------------> Congestion Notification Message <-----------------------------------------| Figure 2. Receiver-Based Notification ]]> Option 2: Direct to Sender Congestion Notification R3 directly sends the congestion information to the traffic source in the ingress DC (DC-A). This approach is considered a "fast" congestion notification because it reduces the feedback path to approximately half-RTT. Upon receiving the notification, the sender can immediately reduce its sending rate for the identified flow. Although the feedback loop is reduced from one-RTT (Option 1) to half-RTT, the absolute latency in a WAN environment remains in the millisecond range. Consequently, the sending rate reduction may still not take effect quickly enough at the congestion node R3, and packet drops due to buffer overflow may still occur. Sending direct congestion notifications from WAN nodes presents two main challenges: Packet Parsing: WAN nodes must parse data packets deeply to extract flow identification information. State Maintenance: For RoCEv2 flows, WAN nodes must obtain and maintain the mapping of source and destination Queue Pairs (QPs) in advance.

+----------+ Congestion +----------+ | Data | +----+ +----+ +----+ | Data | | center A |---| R1 |---| R3 |-=-| R5 |---| center B | +----------+ +----+ +----+ +----+ +----------+ <---------------------| Congestion Notification Message Figure 3. Direct to Sender Notification ]]> Option 3: Coordinated Flow Control and Congestion Control Notification Instead of sending the congestion information directly to the receiver in the egress DC or the source in the egress DC, R3 sends a WAN flow control notification to its upstream node R1 on the data path, notifying R1 to pause the specific flow which caused congestion. If R1 detects local congestion due to the pause of the flow, it can send a flow control notification to its upstream node on the data path. This hop-by-hop notification mechanism continues until the flow control signal reaches the gateway of the ingress DC (DC-A). The DC gateway pauses the flow according to the flow control notification. If this causes local congestion on the DC gateway, it may generate a congestion notification to the traffic source. For RoCEv2 flows, this requires the DC gateway to maintain the source and destination Queue Pair (QP) information. The advantage of this approach is, the control loop latency is comparable to that in Option 2, while it leverages hop-by-hop flow control within the WAN network. By pausing the traffic at each upstream node, the buffer resources on WAN nodes along the data path can absorb burst traffic before the source reduces its sending rate. This mechanism can effectively avoid or alleviate packet loss in the WAN network, even if the rate adjustment at the sender is delayed due to the long distance.

Assume traffic is sent from DC-A (the ingress DC) to DC-B (the egress DC) through the WAN network, taking the path R1-R3-R5. If a traffic burst or surge causes congestion within the egress DC (e.g., on link S1-L2 in Figure 5), the node detecting the congestion (node S1) has the following options for handling it: Option 1: Receiver-Based Congestion Notification (Standard ECN) S1 marks the ECN field in the data packet as "Congestion Experienced" and forwards it to the receiver in the egress DC (DC-B). Upon detecting the ECN mark, the receiver generates a congestion notification message and sends it back to the traffic source in DC-A, instructing the source to reduce its sending rate for the affected flows. This is the standard congestion notification mechanism used in DCN. However, in a DC interconnect scenario, the control loop spans the full WAN RTT, which is typically in the millisecond range. This latency may prevent timely rate reduction. Consequently, Priority-based Flow Control (PFC) may be used within DC-B to prevent S1 from dropping packets while waiting for the sending rate adjustment. If the burst is too large, S1 may still drop packets due to buffer overflow.

DC-B +----------+ +------------+ | | +----+ +----+ +----+ | S1 S2 | | DC-A |---| R1 |---| R3 |---| R5 |---| / \/ \ | | | +----+ +----+ +----+ | L1 L2 L3 | +----------+ +------------+ Congestion Congestion Notification Message <------------------------------------------| Figure 6. Direct to Sender Notification ]]> Option 3: Coordinated Flow Control and Congestion Control Notification This option introduces a coordinated flow control and congestion control mechanism across the DC and WAN networks. When congestion is detected at node S1 within DC-B, it initiates a PFC pause frame towards the DC-B Gateway. This step is critical for immediate packet loss prevention within the DC fabric. The DC-B Gateway receives the PFC pause frame and ensures that no packets are dropped on the S1-L2 link while the congestion signal is being processed and propagated. At the same time, the DC-B Gateway convert the congestion status into an fine-grained flow control signal and injects it into the WAN path at the first hop, R5. Unlike PFC, which is confined to Layer 2, the fine-grained flow control message can traverse the Layer-3 boundaries. Similar to the case described in section 3.1, option 3, the fine-grained flow control propagates hop-by-hop along the path, from R5 to R3, then to R1, and finally reaches the DC-A Gateway. Each WAN node receiving the flow control signal pauses the corresponding flow on its outgoing interface, which effectively throttles the traffic upstream before it reaches the congested point. Upon receiving the fine-grained flow control signal, the DC-A Gateway pauses the flow at the source edge. If this causes local congestion on the DC-A side, the gateway may trigger congestion notification message to the traffic source in DC-A for further rate reduction. For RoCEv2 flows, this requires the DC gateway to maintain source and destination Queue Pair (QP) mappings to ensure precise flow identification. The advantage of this coordinated approach is that, during the propagation of the congestion information to the source in DC-A, DC-B can be protected from packet loss, and the WAN links are protected from consequent congestion and buffer overflow, as reaction to the congestion can be performed at the intermediate WAN and DC nodes respectively.

DC-B +----------+ +------------+ | | +----+ +----+ +----+ | S1 S2 | | DC-A |---| R1 |---| R3 |---| R5 |---| / \/ \ | | | +----+ +----+ +----+ | L1 L2 L3 | +----------+ +------------+ <-----|<-----|<-------|<----- |<------ |<----- WAN WAN WAN WAN DC Congestion Flow Flow Flow Flow Flow Notification Control Control Control Control Control Figure 7. Coordinated Flow Control and Congestion Notification ]]>

Section 3 demonstrates that in WAN-based DC interconnect scenarios, the coordination of flow control and congestion control between the WAN and the DC provides an effective approach to avoid packet loss due to congestion happened in either the WAN or the DCN. This coordination introduces new requirements for fast network notifications for flow control in WAN. The specific requirements are summarized as follows: Fine Granularity: Flow control notifications in the WAN MUST NOT stop all traffic on a specific link or port. The pause signal SHOULD operate at a fine granularity, targeting specific flows, groups of flows, or traffic belonging to a specific tenant, etc. Directed Delivery: Flow control notifications in the WAN MUST NOT be sent via broadcast or multicast. They SHOULD be delivered hop-by-hop along the reverse direction of the data path to upstream nodes, and MAY ultimately reach the ingress node of the specific traffic flows within the WAN. Transport: Since flow control notifications need to be consumed by WAN network nodes in the data plane, it is RECOMMENDED that the notification message be carried within IP packets. Coordination: Flow control notifications in the WAN MAY be generated either upon detecting congestion in the WAN network, or by converting flow control notifications received from downstream DCNs. Additionally, a DC gateway MAY use these notifications as a trigger to send a congestion notification to the source of the traffic flow causing the congestion.

The security considerations in applies to this document. The Flow control notifications generated in WAN MUST be kept in the WAN network domain and MUST NOT be sent directly across the network boundary with DC networks. The fast flow control notification MAY contain nodes and path information of the WAN network, which MUST NOT be exposed to entities outside the WAN domain.

This document does not require any IANA actions.

The authors would like to thank Xuesong Geng and Xingjian He for their review and comments.