RFC6374 Synonymous Flow LabelsFuturewei Technologies Inc.sb@stewartbryant.comSouthend Technical Centerswallow.ietf@gmail.comHuaweimach.chen@huawei.comHuawei Technologiesgiuseppe.fioccola@huawei.comZTE Corp.gregimirsky@gmail.comMPLS Working GroupRFC 6374 describes methods of making loss and delay measurements on
Label Switched Paths (LSPs) primarily as used in MPLS Transport Profile (MPLS-TP)
networks. This document describes a method of extending RFC 6374 performance
measurements from flows carried over MPLS-TP to flows carried over generic MPLS LSPs.
In particular, it extends
the technique to allow loss and delay measurements to be made on multi-point to point
LSPs and introduces some additional techniques to allow more sophisticated
measurements to be made in both MPLS-TP and generic MPLS networks. was originally designed for use as an Operations, Administration, and
Maintenance (OAM) protocol
for use with MPLS Transport Profile (MPLS-TP) LSPs. MPLS-TP only
supports point-to-point and point-to-multi-point LSPs. This document describes
how to use RFC6374 in the generic MPLS case, and also introduces a number
of more sophisticated measurements of applicability to both cases. describes the requirement for introducing
flow identities when using RFC6374 packet Loss Measurements
(LM). In summary RFC6374 uses the loss-measurement (LM) packet as the
packet accounting
demarcation point. Unfortunately this gives rise to a number of
problems that may lead to significant packet accounting errors in
certain situations. For example:Where a flow is subjected to Equal Cost Multi-Path (ECMP)
treatment packets can arrive out of order with respect to the LM
packet.Where a flow is subjected to ECMP treatment, packets can arrive
at different hardware interfaces, thus requiring reception of an
LM packet on one interface to trigger a packet accounting action
on a different interface which may not be co-located with it.
This is a difficult technical problem to address with the
required degree of accuracy.Even where there is no ECMP (for example on RSVP-TE, MPLS-TP LSPs
and pseudowires(PWs)) local processing may be distributed over a number of
processor cores, leading to synchronization problems.Link aggregation techniques may also lead to synchronization
issues.Some forwarder implementations have a long pipeline between
processing a packet and incrementing the associated counter, again
leading to synchronization difficulties.An approach to mitigating these synchronization issue is described in
in which packets are
batched by the sender and each batch is marked in some way such that
adjacent batches can be easily recognized by the receiver.An additional problem arises where the LSP is a multi-point to point
LSP, since MPLS does not include a source address in the packet.
Network management operations require the measurement of packet loss
between a source and destination. It is thus necessary to introduce
some source specific information into the packet to identify packet
batches from a specific source. describes a method of encoding per flow
instructions in an MPLS label stack using a technique called
Synonymous Flow Labels (SFL) in which labels which mimic the
behavior of other labels provide the packet batch identifiers and
enable the per batch packet accounting. This memo specifies how SFLs
are used to perform RFC6374 packet loss and RFC6374 delay measurements.The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”,
“SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “NOT RECOMMENDED”, “MAY”, and
“OPTIONAL” in this document are to be interpreted as described in BCP 14
when, and only when, they appear in all
capitals, as shown here.The data service packets of the flow being instrumented are grouped
into batches, and all the packets within a batch are marked with
the SFL corresponding to that batch.
The sender counts the number of packets in the batch. When the
batch has completed and the sender is confident that all of the
packets in that batch will have been received, the sender issues
an RFC6374 Query message to determine the number actually
received and hence the number of packets lost. The RFC6374
Query message is sent using the same SFL as the corresponding batch of
data service packets. The format of the Query and Response packets is
described in .RFC6374 describes how to measure the packet delay by measuring the
transit time of an RFC6374 packet over an LSP. Such a packet may not
need to be carried over an SFL since the delay over a particular LSP
should be a function of the Traffic Class (TC) bits.However, where SFLs are being used to monitor packet loss or where
label inferred scheduling is used then
the SFL would be REQUIRED to ensure that the RFC6374 packet
which was being used as a proxy for a data service packet experienced
a representative delay. The format of an
RFC6374 packet carried over the LSP using an SFL is shown in .Where it is desired to more thoroughly instrument a packet flow and to
determine the delay of a number of packets it is undesirable to
send a large number of RFC6374 packets acting as a proxy data service
packets (see ). A method of directly measuring the delay characteristics
of a batch of packets is therefore needed.Given the long intervals over which it is necessary to measure packet
loss, it is not necessarily the case that the batch times for the two
measurement types would be identical. Thus, we use a technique that
permits the two measurements are made concurrently and yet relatively
independent from each other. The notion that they are relatively
independent arises from the potential for the two batches to overlap in time,
in which case either the delay batch time will need to be cut short or the loss
time will need to be extended to allow correct reconciliation of the
various counters.The problem is illustrated in below:In case 1 of we show the case where loss measurement alone
is being carried out on the flow under analysis. For illustrative
purposes consider that 10 packets are used in each flow in the
time interval being analyzed.Now consider case 2 of where a small batch of
packets need to be analyzed for delay. These are marked with a different
SFL type indicating that they are to be monitored for both loss
and delay. The SFL=A indicates loss batch A, SFL=D indicates a batch
of packets that are to be instrumented for delay, but SFL D is
synonymous with SFL A, which in turn is synonymous with the underlying
Forwarding Equivalence Class (FEC). Thus, a packet marked D will be accumulated into the A loss
batch, into the delay statistics and will be forwarded as normal.
Whether the packet is actually counted twice (for loss and delay)
or whether the two counters are reconciled during reporting is
a local matter.Now consider case 3 of where a small batch of packets
are marked for delay across a loss batch boundary. These packets
need to be considered as part of batch A or a part of batch B, and
any RFC6374 Query needs to take place after all the packets
A or D (whichever option is chosen) have arrived at the receiving LSR.Now consider case 4 of . Here we have a case where
it is required to take a number of delay measurements within
a batch of packets that we are measuring for loss. To do this
we need two SFLs for delay (C and D) and alternate between
them (on a delay batch by delay batch basis) for the purposes of
measuring the delay characteristics of the different batches of packets.It is possible to construct a large set of overlapping measurement
types, in terms of loss, delay, loss and delay and batch overlap. If
we allow all combinations of cases, this leads to configuration,
testing and implementation complexity and hence increased costs.
The following simplifying rules represent the
default case:Any system that needs to measure delay MUST be able to
measure loss.Any system that is to measure delay MUST be configured to
measure loss. Whether the loss statistics are collected
or not is a local matter.A delay measurement MAY start at any point during a loss measurement
batch, subject to rule 4.A delay measurement interval MUST be short enough that it
will complete before the enclosing loss batch completes.The duration of a second delay (D in batch must be such
that all packets from the packets belonging to a first
delay batch (C in )will have been received before
the second delay batch completes. This condition is satisfied
when the time to send a batch is long compared to the network
propagation time, and is a parameter that can be established
by the network operator.Given that the sender controls both the start and duration of
a loss and a delay packet batch, these rules are readily implemented
in the control plane.A number of methods are described which add to the set of measurements
originally specified in . Each of these methods has different
characteristics and different processing demands on the packet forwarder.
The choice of method will depend on the type of diagnostic that the operator seeks.Three Methods are discussed:Time BucketsClassic Standard DeviationAverage DelayIn this method the receiving LSR measures the inter-packet gap, classifies the
delay into a number of delay buckets and records the number of packets
in each bucket. As an example, if the operator were concerned about packets with
a delay of up to 1us, 2us, 4us, 8us, and over 8us then there would be five buckets
and packets that arrived up to 1us would cause the 1us bucket counter to increase,
between 1us and 2us the 2us bucket counter would increase etc. In practice it
might be better in terms of processing and potential parallelism if, when a packet had
a delay relative to its predecessor of 2us, then both the up to 1us and the 2us counter
were incremented, and any more detailed information was calculated in the analytics
system.This method allows the operator to see more structure in the jitter characteristics
than simply measuring the average jitter, and avoids the complication of needing
to perform a per packet multiply, but will probably need the time intervals between
buckets to be programmable by the operator.The packet format of a Time Bucket Jitter Measurement Message
is shown below:The Version, Flags, Control Code, Message Length, QTF, RTF, RPTF,
Session Identifier, Reserved and DS Fields are as defined in section 3.2
of RFC6374. The remaining fields, which are unsigned integers, are as follows:There will be a number of Interval/Number pairs depending on the
number of buckets being specified by the Querier. If an RFC6374
message is being used to configure the buckets, (i.e. the responder
is creating or modifying the buckets according to the intervals in
the Query message), then the Responder
MUST respond with 0 packets in each bucket until it has been
configured for a full measurement period. This indicates that it was configured
at the time of the last response message, and thus the response
is valid for the whole interval. As per the convention
the Number of pkts in Bucket fields are included in the Query message and set
to zero.Out of band configuration is permitted by this mode of operation.Note this is a departure from the normal fixed format used in
RFC6374.The time bucket jitter measurement message is carried over an LSP in the way described in
and over an LSP with an SFL as described in .In this method, provision is made for reporting the following delay
characteristics:Number of packets in the batch (n).Sum of delays in a batch (S)Maximum Delay.Minimum Delay.Sum of squares of Inter-packet delay (SS).Characteristics 1 and 2 give the mean delay. Measuring the delay of each
pair in the batch is discussed in .Characteristics 3 and 4 give the outliers.Characteristics 1, 2 and 5 can be used to calculate the variance of the
inter-packet gap and hence the standard deviation giving a view of
the distribution of packet delays and hence the jitter. The equation
for the variance (var) is given by:There is some concern over the use of this algorithm for measuring
variance, because SS and S*S/n can be similar numbers, particularly
where variance is low. However the method commends it self by not
requiring a division in the hardware.The packet format of a Multi-Packet Delay Measurement Message
is shown below:The Version, Flags, Control Code, Message Length, QTF, RTF, RPTF,
Session Identifier, Reserved and DS Fields are as defined in section 3.2
of RFC6374. The remaining fields are as follows:The multi-packet delay measurement message is carried over an LSP in the way described in
and over an LSP with an SFL as described in .If detailed packet delay measurement is required then it might be
possible to record the inter-packet gap for each packet pair. In other
than exception cases of slow flows or small batch sizes, this would
create a large (per packet) demand on storage in the instrumentation system,
a large bandwidth to such a storage system and large bandwidth to the analytics
system. Such a measurement technique is outside the scope of this
document.Introduced in is the concept of a one
way delay measurement in which the average time of arrival of a
set of packets is measured. In this approach the packet is time-stamped
at arrival and the Responder returns the sum of the time-stamps
and the number of times-tamps. From this the analytics engine can
determine the mean delay. An alternative model is that the Responder
returns the time stamp of the first and last packet and the
number of packets. This later method has the advantage of allowing the
average delay to be determined at a number of points along the
packet path and allowing the components of the delay to be
characterized. Unless specifically configured otherwise, the
responder may return either or both types of response and
the analytics engine should process the response appropriately.The packet format of an Average Delay Measurement Message
is shown below:The Version, Flags, Control Code, Message Length, QTF, RTF, RPTF,
Session Identifier, and DS Fields are as defined in section 3.2
of RFC6374. The remaining fields are as follows:The average delay measurement message
is carried over an LSP in the way described in
and over an LSP with an SFL as described in .
As is the convention with RFC6374, the Query message contains placeholders
for the Response message. The placeholders are sent as zero.In the discussion so far it has been assumed that we would measure
the delay characteristics of every packet in a delay measurement
interval defined by an SFL of constant color.
In the concept of a sampled
measurement is considered. That is the Responder only measures a packet
at the start of a group of packets being marked for delay measurement
by a particular color, rather than every packet in the marked batch.
A measurement
interval is not defined by the duration of a marked batch of packets
but the interval between a pair of RFC6374 packets taking a readout
of the delay characteristic. This approach has the advantage that
the measurement is not impacted by ECMP effects.This sampled approach may be used if supported by the Responder and
configured by the opertor.We illustrate the packet format of an RFC6374 Query message using SFLs
for the case of an MPLS direct loss measurement in
.The MPLS label stack is exactly the same as that used for the user
data service packets being instrumented except for the inclusion
of the Generic Associated Channel Label (GAL) to allow the receiver to distinguish between
normal data packets and OAM packets. Since the packet loss
measurements are being made on the data service packets,
an RFC6374 direct loss measurement is being made,
and which is indicated by the type field in the ACH (Type = 0x000A).The RFC6374 measurement message consists of the three components,
the RFC6374 fixed-format portion of the message as specified in carried over
the ACH channel type specified the type of measurement being
made (currently: loss, delay or loss and delay) as specified in RFC6374.Two optional TLVs MAY also be carried if needed. The first is the
SFL TLV specified in . This is used to provide the
implementation with a reminder of the SFL that was used to carry the
RFC6374 message. This is needed because a number of MPLS
implementations do not provide the MPLS label stack to the MPLS OAM
handler. This TLV is required if RFC6374 messages are sent over UDP
. This TLV MUST be included unless, by some method outside
the scope of this document, it is known that this information is not
needed by the RFC6374 Responder.The second set of information that may be needed is the return
information that allows the responder send the RFC6374 response to
the Querier. This is not needed if the response is requested in-band
and the MPLS construct being measured is a point to point LSP, but
otherwise MUST be carried. The return address TLV is defined in
and the optional UDP Return Object is defined in .Where a measurement other than an MPLS direct loss measurement is to be
made, the appropriate RFC6374 measurement message is used (for example, one of the
new types defined in this document) and this is indicated to the receiver
by the use of the corresponding ACH type.The RFC6374 SFL TLV is shown in . This contains the
SFL that was carried in the label stack, the FEC that was used to
allocate the SFL and the index into the batch of SLs that were
allocated for the FEC that corresponds to this SFL.Where:This information is needed to allow for operation with hardware that
discards the MPLS label stack before passing the remainder of the
stack to the OAM handler. By providing both the SFL and the FEC plus
index into the array of allocated SFLs a number of implementation
types are supported.This mode of operation is not currently supported by this specification.The inclusion of originating and/or flow information in a packet
provides more identity information and hence potentially degrades the
privacy of the communication. Whilst the inclusion of the additional
granularity does allow greater insight into the flow characteristics
it does not specifically identify which node originated the packet
other than by inspection of the network at the point of ingress, or
inspection of the control protocol packets. This privacy threat may
be mitigated by encrypting the control protocol packets, regularly
changing the synonymous labels and by concurrently using a number of
such labels.The security considerations documented in and
(which in turn calls up and ) are applicable to this
protocol.The issue noted in is a security consideration. There are
no other new security issues associated with the MPLS dataplane. Any
control protocol used to request SFLs will need to ensure the
legitimacy of the request.An attacker that manages to corrupt the RFC6374 SFL TLV could
disrupt the measurements in a way that the RFC6374 responder is unable to
detect. However, the network opertator is likely to notice the
anomalous network performance measurements, and in any case
normal MPLS network security proceedures make this type of attack extremely unlikley.As per the IANA considerations in updated by and , IANA is requested to
allocate the following codeponts in the “MPLS Generalized Associated Channel
(G-ACh) Type” registry, in the “Generic Associated Channel (G-ACh) Parameters”
name space:IANA is requested to allocate a new TLV from the 0-127 range of the
MPLS Loss/Delay Measurement TLV Object Registry in the
“Generic Associated Channel (G-ACh) Parameters” namespace:A value of 4 is recommended.RFC Editor please delete this para The authors thank Benjamin Kaduk and Elwyn Davies for their thorough and thoughtful
review of this document.A Simple Control Protocol for MPLS SFLsIn draft-ietf-mpls-sfl-framework the concept of MPLS synonymous flow labels (SFL) was introduced. This document describes a simple control protocol that runs over an associated control header to request, withdraw, and extend the lifetime of such labels. It is not the only control protocol that moght be used to support SFL, but it has the benefit of being able to be used without modifying of the existing MPLS control prodocols. The existance of this design is not intended to restrict the ability to enhance an existing MPLS control protocol to add a similar capability. A Querier MUST wait a configured time (suggested wait of 60 seconds) before re-attempting a Withdraw request. No more than three Withdraw requests SHOULD be made. These restricctions are to prevent overloading the control plane of the actioning router.Key words for use in RFCs to Indicate Requirement LevelsIn many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements.Ambiguity of Uppercase vs Lowercase in RFC 2119 Key WordsRFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings.MPLS Label Stack EncodingThis document specifies the encoding to be used by an LSR in order to transmit labeled packets on Point-to-Point Protocol (PPP) data links, on LAN data links, and possibly on other data links as well. This document also specifies rules and procedures for processing the various fields of the label stack encoding. [STANDARDS-TRACK]UDP Return Path for Packet Loss and Delay Measurement for MPLS NetworksRFC 6374 defines a protocol for Packet Loss and Delay Measurement for MPLS networks (MPLS-PLDM). This document specifies the procedures to be used when sending and processing out-of-band MPLS performance management Responses over an UDP/IP return path.MPLS Generic Associated ChannelThis document generalizes the applicability of the pseudowire (PW) Associated Channel Header (ACH), enabling the realization of a control channel associated to MPLS Label Switched Paths (LSPs) and MPLS Sections in addition to MPLS pseudowires. In order to identify the presence of this Associated Channel Header in the label stack, this document also assigns one of the reserved MPLS label values to the Generic Associated Channel Label (GAL), to be used as a label based exception mechanism.Packet Loss and Delay Measurement for MPLS NetworksMany service provider service level agreements (SLAs) depend on the ability to measure and monitor performance metrics for packet loss and one-way and two-way delay, as well as related metrics such as delay variation and channel throughput. This measurement capability also provides operators with greater visibility into the performance characteristics of their networks, thereby facilitating planning, troubleshooting, and network performance evaluation. This document specifies protocol mechanisms to enable the efficient and accurate measurement of these performance metrics in MPLS networks. [STANDARDS-TRACK]Synonymous Flow Label FrameworkRFC 8372 ("MPLS Flow Identification Considerations") describes the requirement for introducing flow identities within the MPLS architecture. This document describes a method of accomplishing this by using a technique called "Synonymous Flow Labels" in which labels that mimic the behavior of other labels provide the identification service. These identifiers can be used to trigger per-flow operations on the packet at the receiving label switching router.LDP SpecificationThe architecture for Multiprotocol Label Switching (MPLS) is described in RFC 3031. A fundamental concept in MPLS is that two Label Switching Routers (LSRs) must agree on the meaning of the labels used to forward traffic between and through them. This common understanding is achieved by using a set of procedures, called a label distribution protocol, by which one LSR informs another of label bindings it has made. This document defines a set of such procedures called LDP (for Label Distribution Protocol) by which LSRs distribute labels to support MPLS forwarding along normally routed paths. [STANDARDS-TRACK]Retiring TLVs from the Associated Channel Header of the MPLS Generic Associated ChannelThe MPLS Generic Associated Channel (G-ACh) is a generalization of the applicability of the pseudowire (PW) Associated Channel Header (ACH). RFC 5586 defines the concept of TLV constructs that can be carried in messages on the G-ACh by placing them in the ACH between the fixed header fields and the G-ACh message. These TLVs are called ACH TLVsNo Associated Channel Type yet defined uses an ACH TLV. Furthermore, it is believed that handling TLVs in hardware introduces significant problems to the fast path, and since G-ACh messages are intended to be processed substantially in hardware, the use of ACH TLVs is undesirable.This document updates RFC 5586 by retiring ACH TLVs and removing the associated registry.Moving Generic Associated Channel (G-ACh) IANA Registries to a New RegistryRFC 5586 generalized the applicability of the pseudowire Associated Channel Header (PW-ACH) into the Generic Associated Channel G-ACh. However, registries and allocations of G-ACh parameters had been distributed throughout different, sometimes unrelated, registries. This document coalesces these into a new "Generic Associated Channel (G-ACh) Parameters" registry under the "Multiprotocol Label Switching Architecture (MPLS)" heading. This document updates RFC 5586.This document also updates RFCs 6374, 6378, 6427, and 6428.MPLS Flow Identification ConsiderationsThis document discusses aspects to consider when developing a solution for MPLS flow identification. The key application that needs this solution is in-band performance monitoring of MPLS flows when MPLS is used to encapsulate user data packets.Alternate-Marking Method for Passive and Hybrid Performance MonitoringThis document describes a method to perform packet loss, delay, and jitter measurements on live traffic. This method is based on an Alternate-Marking (coloring) technique. A report is provided in order to explain an example and show the method applicability. This technology can be applied in various situations, as detailed in this document, and could be considered Passive or Hybrid depending on the application.Multi-Protocol Label Switching (MPLS) Support of Differentiated ServicesThis document defines a flexible solution for support of Differentiated Services (Diff-Serv) over Multi-Protocol Label Switching (MPLS) networks. [STANDARDS-TRACK]A Framework for MPLS in Transport NetworksThis document specifies an architectural framework for the application of Multiprotocol Label Switching (MPLS) to the construction of packet-switched transport networks. It describes a common set of protocol functions -- the MPLS Transport Profile (MPLS-TP) -- that supports the operational models and capabilities typical of such networks, including signaled or explicitly provisioned bidirectional connection-oriented paths, protection and restoration mechanisms, comprehensive Operations, Administration, and Maintenance (OAM) functions, and network operation in the absence of a dynamic control plane or IP forwarding support. Some of these functions are defined in existing MPLS specifications, while others require extensions to existing specifications to meet the requirements of the MPLS-TP.This document defines the subset of the MPLS-TP applicable in general and to point-to-point transport paths. The remaining subset, applicable specifically to point-to-multipoint transport paths, is outside the scope of this document.This document is a product of a joint Internet Engineering Task Force (IETF) / International Telecommunication Union Telecommunication Standardization Sector (ITU-T) effort to include an MPLS Transport Profile within the IETF MPLS and Pseudowire Emulation Edge-to-Edge (PWE3) architectures to support the capabilities and functionalities of a packet transport network as defined by the ITU-T. This document is not an Internet Standards Track specification; it is published for informational purposes.Use of Multipath with MPLS and MPLS Transport Profile (MPLS-TP)Many MPLS implementations have supported multipath techniques, and many MPLS deployments have used multipath techniques, particularly in very high-bandwidth applications, such as provider IP/MPLS core networks. MPLS Transport Profile (MPLS-TP) has strongly discouraged the use of multipath techniques. Some degradation of MPLS-TP Operations, Administration, and Maintenance (OAM) performance cannot be avoided when operating over many types of multipath implementations.Using MPLS Entropy Labels (RFC 6790), MPLS Label Switched Paths (LSPs) can be carried over multipath links while also providing a fully MPLS-TP-compliant server layer for MPLS-TP LSPs. This document describes the means of supporting MPLS as a server layer for MPLS-TP. The use of MPLS-TP LSPs as a server layer for MPLS LSPs is also discussed.Pervasive Monitoring Is an AttackPervasive monitoring is a technical attack that should be mitigated in the design of IETF protocols, where possible.Security Framework for MPLS and GMPLS NetworksThis document provides a security framework for Multiprotocol Label Switching (MPLS) and Generalized Multiprotocol Label Switching (GMPLS) Networks. This document addresses the security aspects that are relevant in the context of MPLS and GMPLS. It describes the security threats, the related defensive techniques, and the mechanisms for detection and reporting. This document emphasizes RSVP-TE and LDP security considerations, as well as inter-AS and inter-provider security considerations for building and maintaining MPLS and GMPLS networks across different domains or different Service Providers. This document is not an Internet Standards Track specification; it is published for informational purposes.