RoCEv2 - Powerful, Flexible for Large-Scale AI Deployments

AI clouds handle large-scale workloads, from generative AI to complex training and inference tasks, all requiring high-throughput, low-latency networking. RoCEv2, leveraging RDMA over Ethernet technology, ensures seamless integration with existing infrastructure while delivering high bandwidth, ultra-low latency, and scalable data transmission for AI cloud environments. As the preferred solution for large-scale distributed AI training and inference clusters, RoCEv2 supports a wide range of applications, including cloud computing, storage, big data, machine learning, and telecommunications.

Capabilities

25.6T Solution

51.2T Solution

Benefits

Portfolio

Optimized for Hyperscale AI, Training, and Inference

RoCEv2 merges RDMA’s efficiency with Ethernet’s flexibility, offering seamless performance with lossless transmission, high bandwidth, and ultra-low latency, all at a cost-effective scale.

Seamless Ethernet Integration & Scalable AI Networking

RoCEv2 integrates effortlessly with existing Ethernet infrastructures, lowering deployment costs and simplifying adoption with familiar management tools. Its efficient scalability supports large network deployments, while maximizing bandwidth to ensure AI/ML workloads can process massive datasets without bottlenecks, accelerating deep learning model training and inference.

Openness and Broad Ecosystem

Built on open Ethernet standards, It benefits from a mature ecosystem and a diverse supplier base, including Arista, NVIDIA, Cisco, and Juniper. Its hardware and software compatibility provides greater flexibility in network design, while a well-established supply chain ensures component availability and minimizes deployment risks, making it a resilient and scalable solution for AI infrastructure.

Efficient Resource Utilization with Reduced CPU Overhead

It optimizes resource efficiency through Kernel Bypass and Workload Offloading, shifting data processing tasks from the CPU to RDMA-enabled NICs. This reduces computational overhead, freeing CPU resources for AI workloads while minimizing latency. By bypassing traditional networking layers, RoCEv2 achieves microsecond-level latency, accelerating AI model inference and real-time decision-making.

Lossless Transmission and Intelligent Traffic Management

RoCEv2 ensures lossless Ethernet transmission with PFC (Priority Flow Control) and ECN (Explicit Congestion Notification), preventing packet loss and optimizing traffic flow. QoS (Quality of Service) further enhances performance by prioritizing critical AI workloads, ensuring consistent bandwidth and low latency even in shared network environments. DCQCN (Data Center Quantized Congestion Notification) dynamically adjusts congestion management, maintaining high throughput for large-scale AI training and inference.

Multi-Tenant Isolation for AI Cloud

RoCEv2 delivers strong network isolation, allowing multiple AI workloads or tenants to operate efficiently without interference. This ensures predictable, high-performance networking even in multi-user cloud environments, making it ideal for AI-driven cloud computing. Its ability to provide consistent, dedicated bandwidth enhances both cost efficiency and workload reliability in large-scale AI clusters.

Data Streamlining for Deterministic Performance

RoCEv2 uses UDP-based transport to prioritize high-speed, low-latency data transfer, ensuring stable and consistent performance for time-sensitive AI applications. Its deterministic nature minimizes delays, delivering precise and efficient results for critical workloads.

NADDOD RoCE Networking Solutions for AI & Distributed computing

25.6T Network with 56G SerDes

For Mid-Scale Data Centers and Al Clusters

Cost-Effective

Flexible Scalability

Density-Optimized

Al Data Center

In AI training clusters with 25.6Tbps unidirectional switch capacity, the 2-layer Spine-Leaf architecture, with a 1:1 convergence ratio, can support up to 2048 GPU cards across 256 nodes (or 256 servers).

While multiple configurations exist for 25.6Tbps switches and 400G NICs, the following solution is based on a widely-adopted configuration: a 64-port 400GbE QSFP-DD switch delivering 25.6Tbps throughput. Each server in this setup is equipped with 8 NVIDIA H100 GPUs and 8 ConnectX-7 400G OSFP NICs.

Spine-Leaf
- 400G QSFP-DD SR8 400G QSFP-DD SR8 (100m)
- 400G QSFP-DD DR4 400G QSFP-DD DR4 (500m)
- 400G QSFP-DD FR4 400G QSFP-DD FR4 (2km)
- 400G QSFP-DD DAC / AOC (0.5m - 30m)
Leaf-Server
- 400G QSFP-DD SR8 400G OSFP SR8 (50m)
- 400G QSFP-DD SR4 400G OSFP SR4 (50m)
- 400G QSFP-DD DR4 400G OSFP DR4 (500m)
- Compared to 400G OSFP SR4, 400G OSFP SR8 offers lower cost, reduced BER, and lower latency for connections within 100 meters.

Server
- NVIDIA CX-7 400G OSFP NIC
Leaf
- 64 x 400G QSFP-DD Switches:
- NADDOD N9200-64DC
- Arista 7060DX5-64S
- NVIDIA SN5400
- ......
Spine
- 64 x 400G QSFP-DD Switches:
- NADDOD N9200-64DC
- Arista 7060DX5-64S
- NVIDIA SN5400
- ......

For large-scale AI clusters with 51.2Tbps unidirectional switch capacity, a 2-layer Spine-Leaf architecture with a 1:1 convergence ratio supports up to 8192 GPUs across 1024 nodes (or servers). For even larger deployments, Layer 3 networking extends scalability to 32K GPUs, meeting the demands of massive AI workloads.

This solution offers flexibility in both switches and NICs. Available switch options include 64-port 800GbE OSFP, 64-port 800GbE QSFP-DD, and 128-port 400GbE QSFP112, with this setup based on the widely adopted 64-port 800GbE OSFP switch. Each server is configured with 8 NVIDIA H100 GPUs and 8 400G NICs, with options such as NVIDIA ConnectX-7 OSFP and BlueField-3 DPU/SuperNIC QSFP112, ensuring adaptability for diverse network requirements.

51.2T Network with 112G SerDes

For Large-Scale AI Clusters and Distributed Computing

High Throughtput

Ultra-Low Latency

AI Training Boost

32K GPU Scaling

Spine-Leaf
- 800G OSFP 2xSR4 800G OSFP 2xSR4 (50m)
- 800G OSFP 2xDR4 800G OSFP 2xDR4 (500m or 2km)
- 800G OSFP 2xFR4 800G OSFP 2xFR4 (2km)
- 800G OSFP DAC/ACC (0.5m to 3m)
Leaf-Server (NIC: CX-7 400G OSFP)
- 800G OSFP 2xSR4 400G OSFP SR4 (100m)
- 800G OSFP 2xDR4 400G OSFP DR4 (500m)
- 800G OSFP DAC (0.5m to 3m)
Leaf-Server (NIC: 400G BLF-3 DPU/SuperNIC QSFP112)
- 800G OSFP 2xSR4 400G QSFP112 SR4 (100m)
- 800G OSFP 2xDR4 400G QSFP112 DR4 (500m)
- 2x400G OSFP to 2x400G QSFP112 DAC (0.5m to 3m)

Server
- NVIDIA CX-7 400G OSFP NIC
- 400G BlueField-3 DPU/
- SuperNIC QSFP112
Leaf
- 64 x 800G OSFP Switches:
- NADDOD N9500-64OC
- Arista 7060X6-64PE
- NVIDIA SN5600
- ......
Spine
- 64 x 800G OSFP Switches:
- NADDOD N9500-64OC
- Arista 7060X6-64PE
- NVIDIA SN5600
- ......

Fully Tested and Compatible with Top-tier Switches and NICs

NADDOD's interconnections are rigorously tested and fully compatible with leading devices, including 51.2Tbps and 25.6Tbps switches (e.g., Arista 7060X6 series, NVIDIA Spectrum-4 SN5000 series, Cisco Nexus 9300-GX2 series) and 400G/200G NICs (e.g., NVIDIA ConnectX-7, Broadcom BCM957608 Thor 2), ensuring high reliability and seamless operation in complex RoCE environments.

51.2Tbps Switches25.6Tbps Switches

Speed	Brand	Series	Model Number
800G	Arista	7060X6 Series	7060X6-64PE
	NVIDIA	SN5600	920-9N42F-00RI-7C0
	Cisco	Nexus 9300	N9364E-SG2-Q
		Nexus 9300	N9364E-SG2-O
		Catalyst 8500 Series	8501-SYS-MT
	Juniper	QFX5240	QFX5240-64OD
	Juniper	QFX5240	QFX5240-64QD
	DELL	Z9864F-ON	Z9864F-ON
	H3C	S9827	S9827-64E
	H3C	S9827	S9827-64EP
400G	H3C	S9827	S9827-128DH

Common Network Issues Affecting AI Training Efficiency

80% AI Training Interruptions Stem from Network-Side Issues

95% Network Problems Often Linked to Faulty Optical Interconnects

Common Network Issues In RoCE Networking

NADDOD - Maximizing RoCE Performance for AI Workloads

Full Suite End-to-End RoCE Solutions for Seamless, Scalable AI Networking

NADDOD offers a comprehensive suite of RoCE solutions, expertly crafted to address the complexities of diverse networking environments. Our solutions ensure peak performance across AI and supercomputing networks, regardless of whether an IT team is available or the scale of the deployment. Whether you're building from the ground up or upgrading an existing Ethernet network, NADDOD provides highly customizable solutions tailored to cluster size, budget, and scalability requirements.

With a broad portfolio that includes switches, transceivers, cables, and NICs ranging from 100Gb/s to 800Gb/s, NADDOD delivers flexible solutions to meet any data center’s needs. Our products are fully customizable, catering to a wide range of deployment scenarios, multi-device environments, and AI workloads. As the RoCE ecosystem evolves, device compatibility across vendors can pose challenges—NADDOD’s deep expertise in optical interconnects ensures seamless integration and smooth communication.

NADDOD RoCE Product Portfolio for AI Networking

RoCE Transceivers and Cables

800G/400G RoCE connectivity options enable flexible topologies with a variety of transceivers, MPO connectors, AOCs, ACCs, and DACs featuring 1–2 or 1–4 splitter options.

Backward compatibility connects 400Gb/s clusters to existing 200Gb/s or 100Gb/s infrastructures, ensuring seamless scalability and integration.

RoCE Transceivers & Cables

RoCE Adapters/NICs

The NVIDIA ConnectX-7 RoCE network adapter provides single or dual network ports with speeds of up to 400Gb/s, available in multiple form factors to meet diverse deployment needs, and delivers hardware-accelerated networking, storage, security, and manageability services at data center scale for cloud, telecommunications, AI, and enterprise workloads. ConnectX-7 enables organizations to meet their current and future networking needs in both high-bandwidth and high-density environments.

ConnectX-7 Adapters

RoCE Switches

Powered by the Tomahawk 4/5 chips and preloaded with a SONiC-based NOS, these switches deliver a total bidirectional throughput of 51.2T/102.4Tbps and can handle over 21 Billion Packets per Second(Bpps). NADDOD's 25.6T/51.2T switches are designed to meet the performance requirements of high-demand AI/ML workloads, cloud environments, data centers, and HPC networks.

25.6T/51.2T Switches