]> China Telecom

China ruanke@chinatelecom.cn

Huawei

China zhangli344@huawei.com

Huawei

China zhukeyi@huawei.com

Huawei

China ryan.cheung@huawei.com

Routing Area Routing Area Working Group Bandwidth BGP Load-balancing Routing Although operators typically consider redundant capacity during network design, link congestion may still occur under specific scenarios, such as link degradation or traffic bursts. Several congestion mitigation mechanisms have been proposed for intra-domain links, but congestion is not limited to intra-domain links, it can also occur on inter-domain links. Since changes of available bandwidth do not trigger updates to BGP route selection, BGP cannot alleviate inter-domain link congestion automatically, causing packet loss or increased delay. This document defines a BGP-based inter-domain elastic bandwidth-aware routing (BEBR) mechanism, to alleviate inter-domain link congestion. To avoid the dependency on cross-domain deployment, this document proposes both inbound BEBR and outbound BEBR, designed to address inbound and outbound traffic congestion, respectively. Thus, the congestion on both directions of inter-domain links can be mitigated by deploying BEBR on only one side of the inter-domain link.

Introduction Although operators typically reserve redundant capacity during network design, link congestion may still occur under specific scenarios, such as link degradation or traffic bursts. specifies a congestion avoidance mechanism for intra-domain links. It alleviates congestion by dynamically distributing traffic across both the primary shortest path and load-balancing alternate paths, thereby effectively mitigating intra-domain congestion. However, congestion is not limited to intra-domain links; it can also occur on inter-domain links. When inter-domain link degradation or traffic bursts occur on inter-domain links, BGP routes remain unchanged because the BGP peer relationship between the two nodes is not disconnected. The relevant ASBR (AS Border Router) cannot detect the degradation and thus cannot automatically optimize the forwarding path. As a result, traffic congestion occurs on the inter-domain links, causing packet loss or increased delay. To address this issue, this document defines a BGP-based inter-domain elastic bandwidth-aware routing mechanism to alleviate inter-domain link congestion. To avoid the dependency on cross-domain deployment, this document proposes both inbound BEBR and outbound BEBR, designed to address inbound and outbound traffic congestion, respectively. Thus, the congestion on both directions of inter-domain links can be mitigated by deploying BEBR on only one side of the inter-domain link.

Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Terminology

Load-Balancing Alternate Path:

An alternate routing path utilized for load balancing traffic when the primary path experiences congestion.

Congestion Threshold:

A configured bandwidth utilization limit. Traffic distribution among the primary path and load-balancing alternate paths is triggered when the bandwidth utilization of a local link exceeds this threshold.

Post-restore Threshold:

A configured bandwidth utilization limit. The congested node initiates a fallback to the primary path when the local link's bandwidth utilization drops below this threshold, indicating that congestion has subsided.

Local-domain Peer AS Border Router(LP-ASBR):

An ASBR is considered an LP-ASBR of another ASBR if both belong to the same domain and connect to the same adjacent domain.

Use Cases of Inter-domain Link Congestion On the network shown in , node A and node B on the MAN (metropolitan area network) respectively establish EBGP peer relationships with node C and node D on the backbone network through loopback interfaces. Multiple physical links exist between the EBGP peers, and routes of the MAN and backbone network are advertised to each other through EBGP. Traffic is load balanced between domains. If some physical links between node A and node C fail or traffic bursts occur on some links between the two nodes, BGP routes remain unchanged because the EBGP peer relationship between the two nodes is not disconnected. In this case, the backbone network and MAN cannot detect the faults and thus cannot automatically optimize the forwarding path. As a result, traffic congestion occurs on node A and node C, causing packet loss or increased delay.

Example topology for inter-domain link congestion

Inbound BEBR

Overview of Inbound BEBR Inbound BEBR migrates traffic by adjusting the attributes of advertised BGP routes. It is applicable in scenarios where an ASBR’s inbound link is congested, shows a typical example topology. The process of Inbound BEBR is as follows:

1. Nodes enabling inbound BEBR (e.g., node A and B in ) need to collect the available bandwidth, bandwidth utilization of their inter-domain links, and the volume of each flow (e.g., based on prefix) on inter-domain links. The available bandwidth and bandwidth utilization information of inter-domain links are advertised to their LP-ASBRs (e.g., through BGP-LS).
2. When a node enabling inbound BEBR detects congestion on a local inter-domain link (e.g., link C-A in ) and perceives that sufficient bandwidth is available on alternative inter-domain links(e.g, link D-B in ), it adjusts the BGP attributes of specific routes (e.g., via BGP MED or AS_PATH). This adjustment causes nodes in the adjacent domain to select BGP routes with alternative inter-domain links, effectively diverting traffic from the local congested link.
3. When the congestion on the local inter-domain link (e.g., link C-A in ) disappears, the BGP attributes of specific routes are restored. The updated BGP routes with restored attributes are advertised to ASBRs in the adjacent domain. This update causes nodes in the adjacent domain to select the original routes, thereby reverting traffic back to the local inter-domain link.

Example topology for inbound BEBR

Monitoring and Advertising Link Bandwidth Information Nodes deploying inbound BEBR need to continuously monitor the bandwidth utilization and available bandwidth of their inbound links. Bandwidth utilization is a key factor in determining whether an link is congested. This determination relies on a configurable Congestion Threshold. If the bandwidth utilization of a link exceeds the Congestion threshold, then the link is congested. Additionally, the volume of each flow (e.g., based on prefix) is required to be collected. This information is used to identify which flows should be diverted from the congested link when congestion occurs. The mechanism for obtaining per-flow volume is out of scope for this document. Upon detecting local link congestion, adjusting traffic arbitrarily may cause new congestion on other inter-domain links. Therefore, it is crucial to obtain information regarding the available bandwidth and bandwidth utilization of alternate inter-domain links in the network. The bandwidth information of an ASBR's inter-domain link is required to be advertised to its LP-ASBRs. This can be realized by enabling BGP-LS between an ASBR and its LP-ASBRs, and by enabling BGP EPE (Egress Peer Engineering) between an ASBR and its neighber ASBRs in adjacent domains. The triggers for bandwidth information advertisement are introduced in .

Traffic Adjustment Upon Congestion Once a node deploying inbound BEBR detects that one of its links is expericing congestion, the traffic adjustment mechanism is triggered to divert specific flows from the congested link. The traffic adjustment process is as follows:

1. The congested node (a node that has an inbound inter-domain link experiencing congestion, e.g., node A in ) generates optimization policies and corresponding RPD routing policies (optional). The optimization policies determine which flows need to be diverted from the congested link based on the volume of each flow and the available bandwidth of other alternate inter-domain links.
2. The congested node applies the generated optimization policies to adjust the BGP attributes of specific routes (e.g., by increasing the MED values or AS_PATH lengths of specific routes).
3. Optionally, the congested node sends the RPD routing policies to its LP-ASBRs (e.g., node B in ). The content of the RPD routing policies is to instruct the adjusting of BGP attributes of specific routes advertised by its LP-ASBRs.
4. Optionally, upon receiving the RPD routing policies, the LP-ASBRs (e.g., node B in ) adjust the BGP attributes of specific routes in accordance with the RPD routes.
5. Upon receiving the BGP routes with updated attributes, nodes in the adjacent domain(e.g., node C and D in ) select the routes advertised by the LP-ASBRs(e.g., node B in ) and forward traffic to them, thereby implementing traffic migration.

RPD routing policies are advertised only when the BGP attributes of specific routes advertised by LP-ASBRs need to be adjusted. This mechanism helps to either avoid or preferentially select routes from specific LP-ASBRs during traffic adjustment. To prevent the confliction of simultaneous traffic adjustment from multiple ASBRs, the ASBR with the smallest router ID will execute traffic adjustment when multiple ASBRs are doing traffic adjustment at the same time. Policies may be configured to determine which flows (e.g., based on traffic class or IP prefixes) should be diverted to the alternate inter-domain links.

Traffic Fallback of Inbound BEBR Traffic Fallback refers to the process where an ASBR reverts traffic from the alternate inter-domain link to the original inter-domain link, thereby restoring the network to its original state. The conditions for triggering traffic fallback are as follows:

• Condition 1: After all traffic has reverted to the original inter-domain link, its bandwidth utilization does not exceed the Post-restore Threshold.
• Condition 2: Condition 1 is met and has sustained for a configurable Restore Period (to prevent frequent traffic oscillation).

The traffic fallback process is as follows:

1. The congestion recovery node (the node that previously experienced congestion, e.g., node A in ) cancels the optimization policies, restoring the BGP attributes of affected routes to their original values. The updated BGP routes with restored attributes are advertised to ASBRs in the adjacent domain.
2. Optionally, the congestion recovery node withdraws the RPD routing policies previously sent to its LP-ASBRs (e.g., node B in ).
3. Optionally, upon receiving the RPD routing policy withdrawal, the LP-ASBRs restores the BGP attributes of specific routes and advetises the updated BGP routes with restored attributes to ASBRs in the adjacent domain.
4. Upon receiving the updated routes, nodes in the adjacent domain (e.g., node C and D in ) select routes from the original ASBR, thereby completing the traffic fallback.

Outbound BEBR

Overview of Outbound BEBR Outbound BEBR implements traffic load balancing by steering specific flows to pre-computed load-balancing alternate paths via SR strict path. It is applicable in scenarios where an ASBR’s outbound link is congested, as shown in ). The process of outbound BEBR is as follows:

1. Nodes enabling outbound BEBR (e.g., node A in ) monitor the available bandwidth, bandwidth utilization of their outbound links, and per-flow volume (e.g., based on prefix) forwarded through inter-domain links. The available bandwidth and bandwidth utilization information of the links are advertised to their LP-ASBRs (e.g., via BGP-LS).
2. Nodes enabling outbound BEBR pre-calculate load-balancing alternate paths that bypass the congested inter-domain link.
3. When a node enabling outbound BEBR detects congestion on a local inter-domain link (e.g., link C-A in ) , it migrates specific flows to the pre-computed load-balancing alternate paths via SR strict path, thereby reducing the traffic load on the congested link.
4. When the congestion on the local inter-domain link disappears, the traffic is reverted back to the original local inter-domain link (e.g., link A-C in ).

Example topology for outbound BEBR | +-----+ --------| A |========================| C |-------- +-----+ | Congestion | +-----+ || | | || | | || | | MAN ||IBGP | EBGP | Backbone network || | | || | | || | Traffic | +-----+ | ----------> | +-----+ --------| B |========================| D |-------- +-----+ | | +-----+ -------------------+ +------------------- ]]>

Monitoring and Advertising of Link Bandwidth Information The procedures for monitoring and advertising link bandwidth information are the same as . The difference between outbound BEBR and inbound BEBR is that the nodes enabling outbound BEBR need to monitor and advertise the bandwidth information for their outbound links. In addition, the ASBR and its LP-ASBRs need to be directly connected, so that the ASBRs can collect the bandwidth information of links between the ASBRs and their LP-ASBRs (e.g. link A-B in ).

Load-balancing Alternate Path Calculation The load-balancing alternate paths in BEBR are used for load balancing when the inter-domain link of the primary path is congested. Each BEBR-enabled node SHOULD calculate load-balancing alternate paths to the adjacent domain in advance. The destination of a load-balancing alternate path does not need to align with the primary path, it only needs to be one of the reachable ASBRs of the same adjacent domain. SIDs are required to be configured on inter-domain links (e.g., link A-C and B-D in ) for composing the load-balancing alternate path. IGP is required to be deployed on ASBRs to advertise the SID information to their LP-ASBRs. The algorithm used for load-balancing alternate path calculation is implementation-specific.

Traffic Distribution Upon Congestion Once a node enabling outbound BEBR detects that one of its links is expericing congestion, the traffic distribution mechanism is triggered to distribute traffic between the primary and load-balancing alternate paths. UCMP (Unequal Cost Multi-Path) is recommended for flow distribution. The weight of each path is determined by its available bandwidth and bandwidth utilization. Specifically:

• Path Available Bandwidth: Defined as the minimum available bandwidth among all links on the path (i.e., the bottleneck).
• Path Utilization: Defined as the maximum utilization ratio among all links on the path.

Path weights remain static once determined, unless new congestion is detected on the links of the load-balancing alternate paths. Flows steered to load-balancing alternate paths are forwarded by SR strict paths, ensuring loop-free forwarding. The SIDs of the inter-domain links May be used as the destination for outer encapsulation. The encapsulation method depends on the data plane type:

• SR-MPLS: Packets are encapsulated with an ordered list of MPLS labels representing the load-balancing alternate path.
• SRv6: Packets are encapsulated with an outer IPv6 header and a Segment Routing Header (SRH) containing the SID list for the load-balancing alternate path.

Policies may be configured to determine which flows (e.g., based on traffic class or IP prefixes) should be migrated from the primary path to the load-balancing alternate paths.

Traffic Fallback of Outbound BEBR The conditions for traffic fallback of outbound BEBR are the same as those defined in . The fallback process for outbound traffic is straightforward. The congestion recovery node disables UCMP load balancing on the primary and load-balancing alternate paths, reverting traffic to the primary path.

Operational Considerations

Oscillation Suppression Micro-burst traffic or flapping bundle member links may cause frequent changes in link utilization and congestion state, potentially resulting in traffic distribution oscillation. To mitigate this, the following suppression measures should be implemented:

• Congestion detection and fallback decisions should account for the statistical characteristics of bandwidth utilization over a sustained period, rather than relying solely on instantaneous utilization.
• The Congestion Threshold should be sufficiently separated from the Post-restore Threshold to prevent congestion state flapping caused by minor traffic fluctuations.

Secondary Congestion Handling Although available bandwidth and bandwidth utilization of load-balancing alternate path are considered during traffic distribution, congestion may still occur on the alternate inter-domain links due to unexpected factors. There are two mechanisms for inbound and outbound BEBR to alleviate the potential secondary congestions.

• Inbound BEBR: When a node which initiated traffic adjustment detects that the bandwidth utilization of one of the in-use inter-domain link exceeds the Congestion Threshold, it may adjust the BGP attributes of specific routes to shift traffic away from the congested inter-domain link and alleviate congestion.
• Outbound BEBR: Similarly, when a node which initiated the traffic distribution detects that the bandwidth utilization of one of the in-use load-balancing alternate path exceeds the Congestion Threshold, it may decrease the load-balancing weight of that path to reduce traffic flow on the congested path and alleviate congestion.

In addition to secondary congestion occurring on inter-domain links, such congestion may also arise on intra-domain links. Shifting traffic from one ASBR to another can increase the bandwidth utilization of specific intra-domain links. Therefore, it is recommended that the EBR be deployed within each domain to alleviate congestion on intra-domain links.

Triggers for Bandwidth Information Advertisement BEBR relies on available bandwidth and bandwidth utilization information of inter-domain links advertised by BGP-LS. The frequency of advertisement should be constrained to prevent the control plane overhead. The following recommendations define the triggers for bandwidth information advertisement:

• Minimum Advertisement Interval: The interval between consecutive advertisements must not be less than the configured Minimum Advertisement Interval. The Minimum Advertisement Interval is configurable, a value of 30 seconds is recommended.
• Absolute Change Threshold: If the Minimum Advertisement Interval has elapsed and the available bandwidth has changed by at least x% since the last advertisement, a new advertisement SHOULD be sent. The threshold x is configurable; a value of 5% is recommended.

Applicability The mechanism proposed in this document is applicable to the network scenarios where the following conditions are met:

• Two adjacent domains are directly connected via links.
• Multiple inter-domain links exist between these two adjacent domains.

Compatibility BEBR supports incremental deployment in a network. It does not require coordination with the peer nodes in adjacent domains and can alleviate inter-domain congestion by being deployed on one side only.

Implementation Requirements

Protocol List

Protocols Used in Inbound BEBR The following protocols are used in outbound BEBR:

• BGP
• BGP-LS
• BGP-EPE
• BGP-RPD (Optional)

Protocols Used in Outbound BEBR The following protocols are used in outbound BEBR:

• BGP
• SRv6 /SR-MPLS
• BGP-LS
• BGP-EPE

IANA Considerations This document has no IANA actions.

Security Considerations All security considerations for base BGP-LS and BGP extensions for RPD apply to this document. BEBR relies on bandwidth information advertised via BGP-LS. Incorrect bandwidth information may lead to congestion on specific links. In typical deployments, BEBR is implemented by ASBR within a single domain that is entirely under the control of a single operator. However, if link bandwidth information is transported over unauthenticated channels, a man-in-the-middle attacker could modify the bandwidth values, thereby inducing congestion on specific links. Furthermore, advertising links that are approaching congestion may provide an attacker with valuable information to destabilize the network. For example, an attacker could inject an edge-to-edge flow (i.e., without needing to modify anything inside the network) to trigger congestion on the identified link. Cryptographic authentication mechanisms for link state advertisements can mitigate these risks.

Normative References China Mobile Huawei Huawei Telefonica Huawei IGP normally computes the shortest paths in a network for packet forwarding, without taking the traffic demands and available bandwidth into consideration. When there is a link degradation or partial link failure in a network which causes throughput reduction, or the volume of specific traffic flows increase dramatically, unexpected congestion may happen if only the shortest paths are used for IP forwarding. Conventional centralized Traffic Engineering (TE) focuses on long- term bandwidth and routes planning based on traffic demands, which can not react to the congestions in networks timely. This document describes a distributed path computation and load balancing mechanism named Elastic Bandwidth-aware Routing (EBR), which can alleviate congestions timely before TE finishes the global optimization. It allows IGP-enabled nodes which face congestion to distribute traffic among the shortest paths and load-balancing alternate paths through Segment Routing Traffic Engineering (SR-TE), with weights determined based on the bandwidth utilization and available bandwidth of these paths. It provides an efficient, accurate and backward compatible approach for dynamic link congestion avoidance. This document discusses the Border Gateway Protocol (BGP), which is an inter-Autonomous System routing protocol. The primary function of a BGP speaking system is to exchange network reachability information with other BGP systems. This network reachability information includes information on the list of Autonomous Systems (ASes) that reachability information traverses. This information is sufficient for constructing a graph of AS connectivity for this reachability from which routing loops may be pruned, and, at the AS level, some policy decisions may be enforced. BGP-4 provides a set of mechanisms for supporting Classless Inter-Domain Routing (CIDR). These mechanisms include support for advertising a set of destinations as an IP prefix, and eliminating the concept of network "class" within BGP. BGP-4 also introduces mechanisms that allow aggregation of routes, including aggregation of AS paths. This document obsoletes RFC 1771. [STANDARDS-TRACK] In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. The BGP MULTI_EXIT_DISC (MED) attribute provides a mechanism for BGP speakers to convey to an adjacent AS the optimal entry point into the local AS. While BGP MEDs function correctly in many scenarios, a number of issues may arise when utilizing MEDs in dynamic or complex topologies. This document discusses implementation and deployment considerations regarding BGP MEDs and provides information with which implementers and network operators should be familiar. This memo provides information for the Internet community. In many environments, a component external to a network is called upon to perform computations based on the network topology and the current state of the connections within the network, including Traffic Engineering (TE) information. This is information typically distributed by IGP routing protocols within the network. This document describes a mechanism by which link-state and TE information can be collected from networks and shared with external components using the BGP routing protocol. This is achieved using a BGP Network Layer Reachability Information (NLRI) encoding format. The mechanism applies to physical and virtual (e.g., tunnel) IGP links. The mechanism described is subject to policy control. Applications of this technique include Application-Layer Traffic Optimization (ALTO) servers and Path Computation Elements (PCEs). This document obsoletes RFC 7752 by completely replacing that document. It makes some small changes and clarifications to the previous specification. This document also obsoletes RFC 9029 by incorporating the updates that it made to RFC 7752. A node steers a packet through a controlled set of instructions, called segments, by prepending the packet with a list of segment identifiers (SIDs). A segment can represent any instruction, topological or service based. SR segments allow steering a flow through any topological path and service chain while maintaining per-flow state only at the ingress node of the SR domain. This document describes an extension to Border Gateway Protocol - Link State (BGP-LS) for advertisement of BGP Peering Segments along with their BGP peering node information so that efficient BGP Egress Peer Engineering (EPE) policies and strategies can be computed based on Segment Routing. Huawei China Telecom Co., Ltd. China Telecom Co., Ltd. Verizon Inc. Futurewei Huawei It is hard to adjust traffic in a traditional IP network from time to time through manual configurations. It is desirable to have a mechanism for setting up routing policies, which adjusts traffic automatically. This document describes BGP Extensions for Routing Policy Distribution (BGP RPD) to support this with a controller. Segment Routing (SR) leverages source routing. A node steers a packet through a controlled set of instructions, called segments, by prepending the packet with an SR header. A segment can represent any instruction, topological or service based. SR allows for the enforcement of a flow through any topological path while maintaining per-flow state only at the ingress node of the SR domain. The Segment Routing architecture can be directly applied to the MPLS data plane with no change on the forwarding plane. It requires a minor extension to the existing link-state routing protocols. This document illustrates the application of Segment Routing to solve the BGP Egress Peer Engineering (BGP-EPE) requirement. The SR-based BGP-EPE solution allows a centralized (Software-Defined Networking, or SDN) controller to program any egress peer policy at ingress border routers or at hosts within the domain. The Segment Routing over IPv6 (SRv6) Network Programming framework enables a network operator or an application to specify a packet processing program by encoding a sequence of instructions in the IPv6 packet header. Each instruction is implemented on one or several nodes in the network and identified by an SRv6 Segment Identifier in the packet. This document defines the SRv6 Network Programming concept and specifies the base set of SRv6 behaviors that enables the creation of interoperable overlays with underlay optimization. Segment Routing (SR) leverages the source-routing paradigm. A node steers a packet through a controlled set of instructions, called segments, by prepending the packet with an SR header. In the MPLS data plane, the SR header is instantiated through a label stack. This document specifies the forwarding behavior to allow instantiating SR over the MPLS data plane (SR-MPLS).

Acknowledgements TBD

Contributors Huawei

China zhangka@huawei.com

Huawei

China jie.dong@huawei.com

Huawei

China rainsword.wang@huawei.com

Huawei

China shenming2@huawei.com

Huawei

China jack.gejun@huawei.com

Huawei

China wangyifan82@huawei.com

Huawei

China dingxiangfeng@huawei.com