A Unified Bitmap Framework for In-Network Aggregation and Multicast

Optimizing the Data Center Networks (DCN) is critical for improving the efficiency of AI computing, especially in the scenarios of parallel jobs and multiple tenants. In-Network Computing (INC), an emerging computing paradigm, aims to engage network switches to execute application functions to improve the application performance or reduce the system cost. Two collective communication patterns dominate modern AI computing and are the focus of this document. The first is AllReduce, the workhorse of data-parallel training, which aggregates arrays from all workers and distributes the result back to them. The second is the dispatch and combine of Mixture-of-Experts (MoE) expert parallelism, which has become a primary scaling technique for large models: each token is sent to a small set of selected experts and the expert outputs are combined back at the source. Although the two patterns appear different, they compose the same two primitives in opposite orders -- one aggregates then multicasts, the other multicasts then aggregates -- and both can be accelerated by the same in-network framework. Beyond the AI collectives, the same framework also serves general network services, of which reliable multicast is presented as a third use case. This document defines a single in-network framework and applies it to three use cases. Section 2 describes the common framework: a tree encoded by a bitmap and exercised in an aggregation direction and a multicast direction that share the same bitmap encoding. Section 3 presents the first use case, AllReduce, which aggregates first and multicasts the result second. Section 4 presents the second use case, MoE expert-parallel dispatch and combine, which multicasts first and aggregates second. Neither of the first two use cases is primary; each exercises the same framework in the opposite order. Section 5 presents the third use case, reliable multicast, in which the acknowledgments of all receivers are aggregated in the network and the aggregation tree can be established in either of the two ways demonstrated by the first two use cases. Section 6 examines why the base BIER architecture alone is insufficient and identifies the header extensions the framework requires, and Section 7 discusses the efficiency of the endpoint-set encoding at scale and the trade-offs among alternative encodings.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

The framework represents the tree of a collective operation by a bitmap. Each participating endpoint -- a worker in AllReduce, an expert in MoE, or a receiver in reliable multicast -- is assigned a bit position, in a manner similar to the BFR-id encoding of BIER . A set of endpoints is therefore a bitmap, and a tree over those endpoints is described by the bitmaps registered at, or carried through, its nodes. The same bitmap encoding is used regardless of the direction in which data moves. The tree is exercised in two directions. In the aggregation direction, the leaves send packets toward the root, and each internal node reduces (e.g., sums) the packets received from its children before forwarding the partial result upward. Each aggregating node holds a node bitmap registering the set of leaves it is responsible for, and each packet carries a packet bitmap (P-BM) identifying the leaves whose data it carries; the node completes its aggregation when the bit-wise OR of the received P-BMs equals its node bitmap. In the multicast direction, a packet carries a bitmap of its destinations and is replicated hop by hop by a bitmap-driven forwarding mechanism similar to that of BIER , which requires neither a pre-built tree nor per-flow state in intermediate nodes. Because both directions share the same bitmap encoding, a single bitmap-encoded tree can drive both. In particular, a node bitmap used on the aggregation side can be used directly as the multicast bitmap on the multicast side. The use cases in this document exercise the two directions in opposite orders: AllReduce (Section 3) aggregates first and multicasts second, while MoE dispatch and combine (Section 4) multicasts first and aggregates second. A third use case, reliable multicast (Section 5), follows the multicast-then-aggregate order of MoE but admits both tree-establishment methods -- pre-configuration as in AllReduce or self-installation as in MoE. Its packets carry the framework's bitmap in both directions just as in the first two use cases -- the destination bitmap on the data, the provenance bitmap on the acknowledgments -- with the sole difference that the acknowledgment payload is empty: the bitmap itself is the acknowledgment. The bitmap is chosen as the unifying encoding for the following reasons. First, it is compact: a fixed-length field encodes an arbitrary subset of the endpoints, whatever its size, and simultaneously serves as the destination set, the provenance set, and the per-node tree state, so no separate tree or group representation is needed. Second, it is hardware friendly: the field sits at a fixed offset with a fixed length, and every operation the framework performs on it -- membership (AND), merging (OR), and the completion test (equality) -- maps directly onto wide bit-wise logic that switch pipelines execute at line rate, a capability that BIER deployments have already motivated in switch hardware. Third, it enables stateless multicast: the packet itself carries the complete destination set, so replication requires neither a pre-built distribution tree nor per-flow state at intermediate nodes, while the aggregation side needs only one registered bitmap per node per job, or transient per-operation bitmaps where the tree is self-installed. Fourth, it supports highly dynamic environments: since the target set is expressed per packet, it can change on every packet without any signaling or state update in the network, which is essential for MoE dispatch where virtually every token selects a different set of experts, and convenient for elastic jobs where workers join or leave between iterations. Fifth, the encoding is exact and self-describing: the P-BM of a packet records precisely which contributions its payload carries, so the completion test is an exact set comparison rather than a fragile counter, the merge is idempotent (OR-ing a duplicate P-BM changes nothing, so duplicate membership can never over-complete an aggregation), and a result emitted on a completion timeout identifies by itself which contributions are missing, enabling precise straggler identification and selective retransmission instead of a wholesale restart. Sixth, the encoding is direction agnostic: the same bitmap has a valid reading in both data movement directions, so one bitmap-encoded tree drives an entire collective; configuration and state are not duplicated per direction, and in the MoE case the combine tree is even installed automatically as a by-product of the dispatch multicast, eliminating any separate tree-construction step. Seventh, the encoding is topology independent: a bit names an endpoint, not a path, so the aggregation tree can be mapped onto arbitrary switches, the dissemination tree is free to take a different shape, and rerouting around a link or node failure changes nothing in the packets or the registered bitmaps.

AllReduce is the workhorse collective of data-parallel training. "Reduce" denotes a sum, product, max, or min over arrays from multiple sources; AllReduce reduces the arrays from all participating workers and distributes the resulting array back to all of them. A host-based implementation uses a logical ring or tree over point-to-point connectivity, typically with a dedicated Parameter Server (PS) that collects the arrays, reduces them, and returns the result by unicast. In-Network Aggregation offloads the reduction to the switches: for each job an aggregation tree is overlaid on the DCN with the workers as leaves, the PS as the root, and switches as the internal aggregating nodes. AllReduce exercises the framework by aggregating toward the root and then multicasting the result back to the workers.

In essence, the in-network aggregation traffic follows a tree structure. While each leaf node sends a packet towards the root, each internal tree node aggregates the packets received from its child nodes. The aggregation result at each internal node continues to be sent toward the root. The root finishes the final aggregation and multicasts the result back to all the leaves. The multicast tree does not need to overlap with the aggregation tree (except the root and leaves). We build a Virtual Aggregation Tree (VAT) on top of the DCN topology. The VAT root can be a switch or a server. The VAT leaves are the server nodes. All other VAT nodes are mapped to arbitrary switches with two constraints: (1) each switch can be mapped by at most one VAT node, and (2) network connectivity exists between any two switches that are mapped to two adjacent VAT nodes. Each server node is assigned a bit in a bitmap. For an AllReduce job, the bits for the selected workers are set to ‘1’. On a VAT, each non-leaf node is configured with a bitmap named A-BM to register the set of leaves it is responsible for aggregation. A-BM covers all the downward leaves of the node. When a worker sends a packet with an array for aggregating to the root, the packet also carries a bitmap named P-BM, in which only the bit corresponding to the worker is set to ‘1’. When a switch mapped to a VAT node for the job receives a data packet, it performs the bit-wise AND operation on A-BM and P-BM. If it results in an all-zero bitmap, it means the packet is not supposed to be aggregated at this switch, so it continues to be forwarded towards the root. Otherwise, the packet is terminated at this switch and the array is buffered for aggregation. Once the switch collects all the arrays that need to be aggregated (i.e., the bit-wise OR of the P-BMs from the aggregated packets equals to the A-BM) and conducts the aggregation, the result packet, which carries a P-BM equal to the A-BM of the switch, is sent towards its parent VAT node. This process repeats until the root finishes the final aggregation. Fig. 1 shows a network and a VAT constructed over it. There are 8 servers in the network. Therefore, the bitmap contains 8 bits and w_i is assigned the i-th bit in the bitmap. We assume the first 4 servers (w1 - w4) are used as workers for an AllReduce job. We decide to use s1 to aggregate the arrays form w1 and w2, use s7 to aggregate the arrays from w3 and w4, and use s6 to aggregate the arrays from s1 and s7. To achieve this, we configure the A-BMs for the job on the involved switches as shown in Fig. 1(a), which leads to the VAT as shown in Fig. 1(b).

(a) Physical Topology and INA Job Allocation ============================================= {s7} [00110000] | .-----------+-----------. | | s5 {s6} [11110000] | | .-----+-----. .-----+-----. | | | | [11000000] | | | {s1} s2 s3 s4 | | | | .--+--. .--+--. .--+--. .--+--. | | | | | | | | [w1] [w2] [w3] [w4] w5 w6 w7 w8 {sN} = INA allocated switch sN = non-allocated switch [wN] = job-allocated node wN = non-allocated node (b) VAT (Virtual Aggregation Tree) ==================================== PS {s6} | .--------+--------. | | {s1} {s7} | | .--+--. .--+--. | | | | [w1] [w2] [w3] [w4] PS = Parameter Server {sN} = INA switch [wN] = worker node The algorithm to construct VATs and the protocol for packet routing and forwarding between VAT nodes are out of the scope of this document.

The most efficient way for result dissemination is through a multicast tree. The multicast tree shares the root and the leaves with the corresponding VAT, but may have different shape. Most existing multicast protocols require building explicit multicast trees and maintaining per-flow state at intermediate nodes. Instead, the BIER architecture demonstrates that a multicast packet can carry a succinct bitmap in its header to identify the targets, eliminating both requirements. The result dissemination therefore adopts a bitmap-driven multicast mechanism similar to BIER: the root plays a role analogous to the BFIR and the leaf nodes analogous to the BFERs. Because the VAT bitmap adopts the same bit-position encoding style, the A-BM configured at the VAT root node can be used directly as the multicast bitmap. Note, however, that the dissemination packets are carried in the INA encapsulation with the extensions identified in , rather than in the base BIER encapsulation.

The Mixture-of-Experts (MoE) architecture has become a dominant approach for scaling large models. In expert parallelism, the experts of an MoE layer are distributed across the workers. For each token, a gating (router) function selects the top-K experts. The token is dispatched to its selected experts, each expert applies its feed-forward computation, and the K expert outputs are combined into a single output by a weighted sum, where the weights are the gating probabilities. The dispatch and combine operations are conventionally realized as two all-to-all collectives and constitute a significant communication bottleneck. Dispatch and combine exercise the framework in the reverse order of AllReduce: MoE performs a dynamic multicast (dispatch) first and an aggregation (combine) second. Because the two phases of an MoE layer share the same endpoints (one token source and the same set of K experts) and traverse the network in opposite directions, a single bitmap-encoded tree can serve both phases. The dispatch multicast tree and the combine aggregation tree are congruent: the branching points of the multicast are exactly the merging points of the aggregation. This is in contrast to the AllReduce use case, where the aggregation tree and the dissemination tree need not overlap. Fig. 2 illustrates the shared tree. Reusing the 8-expert bitmap convention of Fig. 1, a token selects K=4 experts (e2, e3, e6, e7), giving a dispatch bitmap D-BM = [01100110]. Panel (a) shows the dispatch: the source multicasts the token down the tree, and at each branch node the D-BM subset bound downstream is copied into a C-BM (denoted ":="), which is the set of selected experts in that node's subtree. Panel (b) shows the combine: the expert outputs travel up the very same tree, the switches aggregate the partial sums, and the bitmap of each switch now serves as its C-BM. The two panels are mirror images, reflecting that the dispatch multicast tree and the combine aggregation tree are one and the same.

(a) Dispatch: dynamic multicast. At each branch node the D-BM subset bound downstream is copied into a C-BM (K=4: e2,e3,e6,e7) ================================================================== [source] D-BM = [01100110] | (branch: C-BM := [01100110]) | (bitmap multicast) v .---------+---------. | | {s1} {s7} branch nodes: C-BM := [01100000] C-BM := [00000110] copy D-BM subset | | .--+--. .--+--. | | | | v v v v [e2] [e3] [e6] [e7] (b) Combine: only the C-BM-bearing nodes aggregate; the expert outputs travel up the same tree in the reverse direction ================================================================== [e2] [e3] [e6] [e7] expert k returns g_k * f_k(x) | | | | '--+--' '--+--' | | {s1} {s7} aggregate when OR(P-BM)=C-BM C-BM = [01100000] C-BM = [00000110] (partial sums) | | '---------+---------' | v [source] C-BM = [01100110]; combined output [source] = token's home node (multicast source for dispatch, root for combine) {sN} = in-network switch; a branch node on the way down is a C-BM-bearing merge node on the way up [eN] = selected expert (multicast egress for dispatch, leaf for combine) ":=" = C-BM installed by copying the D-BM subset at dispatch time

Each expert (or the worker hosting it) is assigned a bit in a bitmap, in the same bit-position style as the BFR-id encoding of BIER. For a given token, the set of top-K selected experts is encoded as a dispatch bitmap named D-BM, in which exactly the K bits of the selected experts are set to '1'. The token source acts as the multicast ingress (analogous to a BFIR) and the K experts act as the egresses (analogous to BFERs). The token packet carrying the D-BM is multicast to the K experts directly by the bitmap-driven forwarding of the framework. Unlike a conventional multicast group, the target set in MoE is highly dynamic: virtually every token selects a different combination of experts. The stateless, per-packet bitmap-driven forwarding, pioneered by BIER, is therefore an excellent match, since it requires neither an explicit multicast tree to be constructed in advance nor per-flow state in intermediate nodes. To enable the subsequent weighted combine without burdening the network nodes with the gating weights, the per-expert gating weight is carried alongside the token in the dispatch packet, so that each expert can pre-multiply its output by its own weight. The combine state is installed as a side effect of the dispatch. At each node where the dispatch packet is replicated to more than one next hop, the node copies the D-BM carried in the packet into a per-token combine bitmap, C-BM, and retains it for the duration of the operation. More precisely, each replicated copy carries only the subset of the D-BM bound for the corresponding next hop, and the node's C-BM is the union of those subsets, i.e., the set of selected experts reachable through that node. A node that forwards the dispatch packet to a single next hop is not a branch node and installs no C-BM. Consequently, the combine tree is established automatically from the dispatch multicast, with no separate tree construction.

After computation, each expert sends its (weight-applied) output back toward the token source for combination. Because the experts pre-multiply their outputs by the gating weights, the combine reduces to a plain summation, which is identical to the reduction primitive used in the AllReduce aggregation phase; the in-network nodes need not be aware of the weights. The aggregation follows the same bitmap procedure as the AllReduce aggregation phase, and is driven entirely by the C-BM that was installed at the branch nodes during dispatch. Each returning result packet carries a P-BM with only the bit of its originating expert set. Only a node that has a C-BM configured for the token participates in the aggregation: it terminates and buffers the result packets whose experts belong to its C-BM, and once the bit-wise OR of the collected P-BMs equals its C-BM, it performs the summation and forwards a single result packet carrying a P-BM equal to its C-BM toward the source. A node that has no C-BM for the token performs no aggregation and simply forwards the result packet upward. This repeats until the source obtains the fully combined output for the token. Because the C-BM is the D-BM copied at the dispatch branch nodes, the combine tree is exactly the reverse of the dispatch multicast tree and requires no pre-configuration as a VAT would. The switches that replicated the token on the way down are precisely the switches that merge the expert outputs on the way up.

Constructing a multicast-and-aggregation tree per individual token incurs prohibitive overhead. In practice, tokens are grouped so that those selecting the same set of experts share a D-BM, or the operation is organized at the granularity of the (source, expert) pairs of the underlying all-to-all exchange. Unlike the stateless multicast of the dispatch phase, the combine phase requires transient state at the in-network nodes to buffer partial sums together with the C-BM and a completion timeout. A token or group identifier is carried throughout both phases so that the combine correctly pairs the expert outputs with their originating token. MoE-specific semantics such as expert capacity limits, token dropping, and weight normalization are handled at the end hosts; the in-network nodes perform only the multicast replication and the summation.

Reliable multicast is a long-standing problem in which a source must confirm that every member of a receiver set has received a packet or a block of data. In an AI data center the pattern is common outside the collectives proper: distributing model checkpoints, weights, container images, or configuration to a large set of nodes, where delivery must be verified. The classic obstacle is ACK implosion: if every receiver unicasts an acknowledgment back, the source is hit by one ACK per receiver for every data packet, a burst proportional to the receiver-set size. In-network aggregation of the acknowledgments removes the implosion, and this framework supports it directly: the receiver set of the multicast is a bitmap, and the ACKs are aggregated over a tree against that same bitmap.

The source multicasts a data packet to the receiver set encoded as a bitmap, using the stateless bitmap-driven multicast of the framework. Each receiver returns an acknowledgment that is, in every respect, an ordinary aggregation-direction packet of this framework: it carries the same endpoint-set bitmap field -- a P-BM with only the receiver's own bit set -- together with the sequence number of the acknowledged data (a packet or a window), in the same encapsulation as the contribution packets of the first two use cases. Aggregating nodes apply exactly the same logic as before: buffer, OR the P-BMs, and emit one merged acknowledgment carrying the OR-ed bitmap upward when the OR equals the registered bitmap. The source ultimately receives a single ACK whose P-BM equals the full receiver set. Reliable multicast thus introduces no new packet type and no new node behavior; both directions carry the framework's bitmap just as in AllReduce and MoE -- the destination bitmap on the data, the provenance bitmap on the acknowledgments. Two properties are nevertheless worth noting. First, the acknowledgment's payload is empty: the P-BM it carries is the entire information content, so the aggregation is a pure bit-wise OR with no arithmetic on any payload, no payload buffering, and per-node state of exactly one bitmap and one sequence number per outstanding window. This makes reliable multicast the lightest of the three use cases to implement: any node capable of BIER-style bitstring processing plus minimal transient state can participate. Second, the failure information is directly actionable: if the completion timer expires, the node forwards a partial ACK, and the source computes the non-acknowledging set as the bit-wise AND of the receiver bitmap with the complement of the received P-BM. That complement set is itself a valid multicast bitmap, so the selective retransmission is simply another multicast whose bitmap is the computed miss set -- the repair loop closes entirely within the encoding.

The ACK-aggregation tree can be established in either of the two ways already demonstrated, and the choice is a policy decision, not a protocol change; the ACK processing at every node is identical in both. Pre-configuration, as in the AllReduce use case: for a long-lived multicast group, the control plane maps an aggregation tree onto selected switches and registers at each node an A-BM covering the receivers below it. The setup cost is amortized over the lifetime of the group, and the tree can be engineered (placement, fan-in) independently of the data path. This method suits stable, repeatedly used receiver sets such as a training cluster receiving periodic checkpoint distributions. Self-installation, as in the MoE dispatch/combine use case: the data multicast itself installs the tree. At every branch node where the data packet is replicated toward more than one next hop, the node records a C-BM equal to the union of the bitmap subsets sent downstream, as transient state keyed by the flow and sequence number, with a completion timeout. The returning ACKs are aggregated against these C-BMs over the exact reverse of the path the data actually took. This method requires zero control-plane involvement, works for one-shot or per-packet dynamic receiver sets, and automatically adapts when consecutive packets (for example, retransmissions to a shrinking miss set) take different trees. The two methods compose naturally: a deployment can pre-configure the tree for the initial bulk distribution to the full, stable receiver set, while the selective retransmissions -- whose receiver sets shrink and change per round -- rely on self-installation.

Acknowledgment granularity is a transport policy: per-packet ACKs minimize repair latency, while per-window ACKs amortize the aggregation state; in either case the sequence number carried in both directions pairs an ACK with the data it acknowledges and keeps concurrent windows apart. Since ACKs carry no user data, an aggregating node never buffers payload in this use case; memory pressure is bounded by the number of outstanding windows. Loss of an ACK is indistinguishable from loss of the data at the source and is resolved by the same retransmission, at the cost of a duplicate delivery that the receiver suppresses by sequence number. Note that a receiver acknowledges receipt, not application-level consumption; end-to-end semantics beyond delivery confirmation remain with the endpoints.

The multicast direction of this framework adopts a bit-position encoding and a replication behavior modeled on BIER , and Sections 3 through 5 show that a node bitmap on the aggregation side can be used directly as the multicast bitmap on the multicast side. Nevertheless, the framework cannot be realized by the BIER architecture as specified in alone. BIER defines a stateless, unidirectional multicast replication service; INA additionally requires a stateful aggregation service that runs over the same bitmap encoding but in the opposite direction and with different per-packet semantics. This section enumerates the gaps and the header information that must be defined beyond the base BIER encapsulation, limited to the information that an in-network node requires to correctly execute the replication, aggregation, and forwarding logic.

In BIER, the bitstring carried in a packet has a single meaning: the set of BFERs to which the packet must be replicated. In this framework the same bitmap encoding is carried in two directions with two distinct semantics. In the multicast direction the bitmap (e.g., the result bitmap in AllReduce dissemination, the D-BM in MoE dispatch, or the receiver bitmap of a reliable-multicast data packet) is a destination set and drives replication; in the aggregation direction the bitmap (the P-BM) is a provenance set that identifies the contributions carried in the payload (or, where the payload is empty as in reliable-multicast acknowledgments, the acknowledging receivers themselves) and drives the termination, buffering, and completion test (OR(P-BM) = A-BM or C-BM) at the aggregating nodes. A node receiving a packet must be able to distinguish the two cases unambiguously; otherwise an aggregation-direction packet would be misinterpreted as a replication request. The base BIER header has no notion of direction, so an explicit direction indicator (e.g., a flag or a distinct next-protocol/DSCP-like code point) MUST be carried with every INA packet. The processing model in the aggregation direction also departs from BIER transit behavior in a fundamental way: an aggregating node terminates the packet, buffers and modifies user payload, and later emits a new packet whose P-BM is the union of the consumed ones. This read-modify-merge behavior is outside the BIER forwarding pseudocode and requires per-job state (the A-BM or C-BM) at selected nodes, which pure BIER explicitly avoids.

BIER scopes a bitstring by sub-domain, bitstring length (BSL), and set identifier, which together identify a bit position space shared by all multicast flows in the domain. INA state, in contrast, is inherently per job: the same bit position may be assigned to different endpoints in different jobs, the A-BM/C-BM installed at a switch is meaningful only within its job, and multi-tenant isolation (see Section 8) requires that bitmaps of different jobs can never be combined. Every INA packet in both directions therefore MUST carry a job (or job plus tenant) identifier that binds the packet to its bit-space namespace and to the aggregation state it is allowed to touch. No equivalent field exists in the base BIER encapsulation.

A collective operation is rarely a single packet exchange. In AllReduce a large array is segmented into chunks that are aggregated independently and possibly in a pipeline, in MoE a stream of tokens (or token groups) traverses the same tree in rapid succession, and in reliable multicast every acknowledgment must be paired with the data packet or window it acknowledges. The aggregating nodes must pair contributions belonging to the same chunk or token before applying the completion test, must keep concurrent chunks apart, and must de-duplicate retransmitted contributions so that a replayed packet is not counted twice in the OR(P-BM) computation. This requires an explicit sequence number (or chunk/token identifier) in every packet of both directions, carried end to end and echoed in the result packets so that the sources can match results to requests. BIER, being a stateless replication service, defines no sequencing; this information is an INA-specific header field.

BIER mandates a flat bitstring of a negotiated BSL. As discussed in , a single flat bitmap is not the most efficient endpoint-set encoding at every operating point: hierarchical bitmaps and explicit node-list encodings are preferable in different regimes of network scale and active-worker density. The INA header therefore MUST carry an encoding-type indicator so that a node can parse the endpoint-set field correctly, and the set of supported encodings is expected to be negotiated per job at setup time. This is a generalization of, rather than a supported option in, the BIER encapsulation.

In summary, an INA encapsulation needs to carry, beyond the endpoint-set encoding itself, exactly the per-packet information that the replication and aggregation logic consults: (1) a direction indicator distinguishing aggregation from multicast, (2) a job/tenant identifier, (3) a sequence or chunk/token identifier, and (4) an encoding-type indicator. The encapsulation SHOULD additionally carry a version field so that it can evolve. Whether these fields are realized as an extension of the BIER header or as a new encapsulation that reuses only the BIER bit-position semantics is a protocol design decision left to a companion document; this document only establishes the functional requirements.

The flat bitmap used in the preceding sections assigns one bit to every endpoint in the domain, so the header field grows linearly with the domain size regardless of how many endpoints actually participate in a job. Modern AI data centers interconnect tens of thousands to hundreds of thousands of accelerator endpoints; a flat bitmap over such a domain is far beyond a practical header budget and beyond the bitstring lengths that switch hardware can match and OR at line rate. The appropriate encoding depends on two parameters: the total domain size N and the number of active endpoints K of the job (the set bits). Two regimes deserve distinct treatment.

When K is large (a dense endpoint set, typical of large training jobs whose AllReduce involves most of the allocated accelerators), the bitmap itself is the efficient representation of the set, and the problem is only its absolute length. Two complementary techniques apply. The first is a hierarchical bitmap. The endpoint space is organized into blocks that follow the physical hierarchy of the DCN (e.g., a block per pod or per rack scope), and the endpoint set is encoded as an upper-level bitmap selecting the blocks that contain active endpoints, plus one lower-level bitmap per selected block. A two-level encoding reduces the worst-case header size from N bits to roughly B + b bits per traversed branch, where B is the number of blocks and b the block size, and it maps naturally onto the aggregation structure: intra-block aggregation trees reduce within a pod using only the lower-level bitmap, and an upper-level tree spanning the block roots reduces across pods using only the upper-level bitmap. The same two-level structure serves the multicast direction, in the spirit of the set/sub-domain partitioning of BIER but arranged so that each level is a self-contained aggregation domain. The second is to construct multiple parallel aggregation/multicast trees, each covering a partition of the endpoint set and each using an independent (and therefore short) bit-space. In the aggregation direction, the roots of the per-partition trees produce partial results that are combined by a second-stage aggregation among the roots; in the multicast direction, one packet copy is injected per partition, analogous to the per-set copies of BIER when a multicast group spans multiple sets. Multiple trees also provide a degree of load balancing and fault isolation, at the cost of a second-stage operation and of coordinating the per-partition completion.

When K is small relative to N (a sparse endpoint set), a bitmap is almost all zeros. The MoE dispatch of Section 4 is the canonical example: a token selects only K = 2 to 8 experts out of a potentially very large expert population, and each token may select a different combination, so the encoding is carried per packet. In this regime an explicit node-list encoding, i.e., a list of K endpoint identifiers of log2(N) bits each (equivalently, the indices of the set bits), is smaller than the bitmap whenever K * log2(N) is below the bitmap length, which for small K holds by a wide margin in absolute terms. The absolute saving, however, must be weighed against the packet size. INA payloads are large: an AllReduce contribution is a gradient chunk that typically fills the MTU, and an MoE dispatch carries a token's hidden-state vector of at least several kilobytes (favoring jumbo frames). Against such payloads, the header overhead is a ratio, not a byte count. For a domain of a few thousand endpoints, a flat bitmap of a few hundred bytes adds only a small percentage to a multi-kilobyte packet, and eliminating it buys little bandwidth while the processing cost of a variable-length list (see below) is paid on every packet at every hop. The node-list encoding is therefore worthwhile only when both conditions hold: the domain is very large, so that a flat bitmap (tens of thousands of bits or more) would consume a non-trivial fraction of even a jumbo-frame packet or exceed the header budget and match width of the switch pipeline, and K is very small, so that the list stays short and its processing bounded. When either condition fails -- a moderate domain, or a dense active set -- the flat or hierarchical bitmap is preferable. The list encoding preserves the semantics of the framework unchanged: the D-BM, P-BM, and C-BM simply enumerate their members instead of setting bits, the C-BM installed at a branch node is the sub-list bound downstream, and the completion test becomes a set-union comparison instead of a bit-wise OR.

The encodings are not equivalent in network processing cost, and the choice is a trade-off between header bandwidth and per-node work. A flat bitmap admits the simplest data-plane implementation: fixed offset, fixed length, and the AND, OR, and equality operations of the aggregation completion test map directly onto wide bit-wise logic at line rate, which is why BIER adopts it. A hierarchical bitmap introduces a variable-length, multi-part field: a node must parse the upper level to locate the lower-level bitmap(s) relevant to it, and a branch node must be able to split and re-assemble the per-block sub-bitmaps when replicating or merging, adding parsing depth and state to the pipeline. A node list is the most compact for sparse sets but the most expensive to process: the field is variable-length, membership tests and set unions replace single-cycle bit-wise operations, and a hardware implementation typically converts the list into an internal (possibly local-scope) bitmap at ingress, bounding the supported K. Sorted lists and a per-job cap on K keep these costs manageable. The decision must therefore weigh the header-overhead ratio against the per-hop processing complexity jointly: the bitmap should be regarded as the default, and the node list as a special-case optimization for the very-large-domain, very-small-K corner described above. Because no single encoding is optimal across regimes, the encoding style is expected to be selected per job (or, for MoE dispatch, per packet class) at setup time, signaled by the encoding-type indicator required in . A deployment may also translate between encodings at domain boundaries, e.g., carrying a node list from the host and converting it to a local bitmap inside a pod-scale aggregation domain, so that the header efficiency of the list and the processing efficiency of the bitmap are obtained where each matters most.

Security Considerations In-Network Aggregation changes the trust model of a collective operation in two ways: the switches read and modify user data (gradients in AllReduce; tokens, gating weights, and expert outputs in MoE), and packet forwarding is driven by a bitmap that an endpoint can influence. The considerations below therefore assume that INA is deployed within a single administrative fabric, and that a control plane authorizes jobs, endpoint bit assignments, and per-job resources. Because the multicast direction adopts a bitmap-driven forwarding modeled on BIER, the security considerations of also apply to it. Confidentiality of in-flight data. In-network reduction requires the switch to operate on plaintext, so aggregated payloads are exposed to, and transiently buffered in, the network devices. Deployments SHOULD confine INA to a trusted fabric, encrypt the host-to-switch links (e.g., MACsec or IPsec), keep aggregation buffers in protected memory, and zeroize them after use. Because every in-network operation defined here is a summation, confidentiality can be preserved by additive masking (secret sharing): the workers add random masks that cancel in the aggregate, so a switch adds only masked values and never sees cleartext, while the root recovers the true sum. Masking is applicable to sum and average reductions but not to max or min, and is far cheaper than homomorphic encryption. Result integrity and poisoning. Because the reduction is performed in the network, there is no end-to-end checksum over the final value, and a faulty or malicious switch can silently corrupt an aggregate. Endpoints SHOULD carry an end-to-end verifier, such as a random linear-combination checksum or a sum of per-source digests, that the root or the workers recompute independently, and SHOULD fall back to host-based reduction on a mismatch. Contribution packets MUST be de-duplicated (for example using sequence numbers together with the P-BM) so that a retransmitted or replayed packet is not counted twice in the OR(P-BM) completion test. Contribution, result, and dissemination packets SHOULD be authenticated with a per-job key (an AEAD or MAC over the headers and a payload digest) so that spoofed or injected packets are rejected; this also protects the MoE gating weights carried in the dispatch packet. Acknowledgment spoofing in reliable multicast. In the reliable multicast use case the switches touch no user payload, but a spoofed or replayed acknowledgment can falsely mark a receiver as acknowledged, suppressing the retransmission and silently masking a delivery failure -- a harm that mirrors result poisoning. Acknowledgment packets MUST therefore be subject to the same source validation, job binding, and sequence-number de-duplication as contribution packets, and SHOULD be authenticated with the per-job key. Bitmap-driven amplification. A packet whose bitmap has many bits set forces heavy replication, and in MoE the bitmap is dynamic and per-packet, so a crafted dispatch packet is a traffic amplification and denial-of-service vector. Ingress nodes SHOULD verify that only a job's authorized source sets bits and only within that job's allowed bit range, SHOULD bound the set-bit count and replication fan-out per packet, and SHOULD rate-limit per source. The multicast scope MUST be enforced at the INA domain boundary. Multi-tenant isolation. Switches hold per-job state (A-BM, and the MoE C-BM soft state) and per-endpoint bit assignments. A job or tenant identifier MUST be carried so that each job has its own bit-space namespace and the A-BM, P-BM, and C-BM of one job can never be combined with those of another. Switch aggregation memory SHOULD be partitioned per job, each packet's job identifier SHOULD be validated against the state it touches, and installation of aggregation state SHOULD be restricted to the control plane. State exhaustion and availability. The MoE combine installs transient per-token (or per-group) C-BM state with a completion timeout at branch nodes, and the aggregation phase buffers partial results. Flooding dispatch or contribution packets can exhaust this memory, and withholding an expected contribution can stall a collective until the timeout expires. Deployments SHOULD enforce per-job quotas on aggregation buffers and soft-state entries with admission control at job setup, use bounded and short completion timeouts with a graceful fallback (forwarding a partial result or reverting to host-based reduction), rate-limit packets per source, and require authentication so that unauthorized traffic cannot create state. The masking and in-network verification mechanisms add overhead and, in the case of masking, apply only to summation-style reductions. Both are therefore OPTIONAL and are expected to be negotiated on a per-job basis.

IANA Considerations This document requires no IANA actions.