ERNIC-CMAC FPGA RDMA Integration

An FPGA networking integration project that connects AMD/Xilinx ERNIC with a CMAC UltraScale+ transmit path on an Alveo U50-class platform.

Maintained by LLYniku.

Important

This repository is a research and bring-up archive, not a turnkey Vivado project or a production RNIC. It contains project-specific integration work together with AMD/Xilinx-generated example-design material. The ERNIC and CMAC IP cores themselves are vendor IP and are not included as portable source deliverables.

Overview

The project explores how to turn the ERNIC example design into a physical CMAC transmit pipeline. Its current RTL connects register configuration, WQE-driven RDMA test traffic, iCRC generation, a 512-bit AXI-Stream datapath, CMAC transmission, board constraints, and on-chip debug.

AXI4-Lite register sequencer
             |
             v
        ERNIC IP core <---- WQE/test generators and response checkers
             |
       512-bit AXI-Stream
             |
      CDC FIFO / iCRC path
             |
   CMAC TX adapter and state machine
             |
      CMAC UltraScale+ IP
             |
        U50 QSFP28 TX

中文简介：本工程基于 AMD/Xilinx ERNIC 与 CMAC UltraScale+ 示例设计，完成 Alveo U50 上的 RDMA 发包链路集成，覆盖 AXI-Lite 寄存器配置、WQE 测试流、 RDMA Send/Read/Write 辅助状态机、iCRC、512-bit AXI-Stream、跨时钟 FIFO、 QSFP28 约束和 ILA 调试。

Project highlights

• Integrates the ernic_0 and cmac_usplus_0 IP blocks around a 512-bit AXI-Stream datapath.
• Provides a 142-entry AXI4-Lite register configuration image and 16 embedded 512-bit WQEs.
• Exercises Send, Send with Invalidate, RDMA Read, RDMA Read Response, RDMA Write, and ACK-oriented test states.
• Adds an iCRC pipeline before traffic is forwarded to the CMAC-side packet adapter.
• Uses xpm_fifo_axis for the retained cross-clock AXI-Stream path.
• Includes U50/QSFP28 clock, GT lane, LED, and HBM trip constraints.
• Exposes GT lock, AXI-Stream handshake, packet-state, valid, and last signals through an ILA integration point.

Main data path

The active board-oriented snapshot is under hardware_test/:

File	Role
cmac_usplus_0_exdes.v	Board top; instantiates CMAC, ERNIC wrapper, packet adapter, clocks, and ILA
ernic_0_exdes_top.v	ERNIC example-design wrapper, register/WQE/checker integration, FIFO, and iCRC path
ernic_0_rnic_exdes_send_rdresp_ack_pkt_gen.v	RDMA Send, Read Response, ACK, and WQE-posting state machine
cmac_usplus_0_axis_pkt_gen.v	CMAC TX control and AXI-Stream forwarding state machine
XRNIC_Reg_Config.vh	Register image, QP/network parameters, WQEs, and test enables
cmac_usplus_0_example_top.xdc	Project-specific clocks, GT lanes, LEDs, and U50 constraints
alveo-u50-xdc.xdc	Vendor U50 master constraint reference

Supporting modules are retained under ernic/:

• reg/: AXI4-Lite transaction and ERNIC register configuration logic;
• wqe/: WQE generation and memory-response emulation;
• crc/: RoCE/iCRC helper pipeline;
• rx/: RDMA packet filter;
• root-level checker modules: TX, RX, write-path, and data-path verification.

Other directories are historical rather than independent releases:

Directory	Meaning
ernic/test/	Simulation-oriented integration snapshot from the RDMA Send stage
ernic/hero/	Consolidated intermediate snapshot with CMAC and ERNIC helpers
cmac/	Earlier CMAC example-design and TX experiments
background/	Original/reference material used during integration
directories ending in copy	Explicit working backups retained by the original project
.history/	Editor-generated point-in-time source snapshots

When studying or rebuilding the design, start with hardware_test/, then resolve its helper modules against ernic/. Do not compile every snapshot at once: several directories intentionally contain modules with the same names.

Configuration snapshot

The checked-in hardware_test/XRNIC_Reg_Config.vh currently defines:

Setting	Retained value
Register transactions	142 address/data pairs
WQE image	16 × 512-bit entries
RDMA Read WQEs	8
RDMA Write WQEs	8
Send test	enabled
RDMA Read test	enabled
RDMA Write test	enabled
AXI-Stream width	512 bits / 64-byte tkeep
CMAC packet parameters	PKT_NUM=1000, PKT_SIZE=522
GT reference clock constraint	161.1328125 MHz (6.206 ns)
Init clock constraint	100 MHz (10 ns)

The register image embeds MAC addresses, IPv4 addresses, QP state, PSNs, memory addresses, and keys. Treat these values as a lab snapshot. Review and regenerate them for your own host, peer, memory map, and queue-pair configuration before programming hardware.

Requirements

You need:

• an AMD/Xilinx Vivado installation that supports your Alveo U50 device;
• access to compatible ERNIC and CMAC UltraScale+ IP licenses/output products;
• the matching U50 board files or exact FPGA part;
• CMAC and ERNIC simulation models for behavioral simulation;
• an external RoCE/RDMA peer, packet capture setup, or loopback environment for protocol-level validation;
• optionally, Vivado Hardware Manager and ILA for board bring-up.

The exact Vivado release and generated IP configuration are not recorded in this repository. The testbench prints CMAC core version 3.1, but that is not enough to identify the original Vivado toolchain.

Reconstructing the Vivado project

This repository does not include .xpr, block-design Tcl, .xci, generated IP output products, or a one-command build. A practical reconstruction flow is:

1. Create a new RTL project for the exact U50 board/part in your lab.

2. Recreate these IP instances with names and interfaces matching the RTL:

   • cmac_usplus_0: CMAC UltraScale+ with the 512-bit AXI-Stream TX interface;
   • ernic_0: ERNIC with 32-bit AXI4-Lite control and the interfaces used by hardware_test/ernic_0_exdes_top.v;
   • ila_0: ten probes matching the widths in hardware_test/cmac_usplus_0_exdes.v.

3. Add only the active hardware_test/ implementation files listed above.

4. Add the required helper RTL from ernic/reg, ernic/wqe, ernic/crc, ernic/rx, and the root-level checker modules.

5. Make the directory containing XRNIC_Reg_Config.vh available on the Verilog include path.

6. Set cmac_usplus_0_exdes as the synthesis top.

7. Add hardware_test/cmac_usplus_0_example_top.xdc; use the master U50 XDC as a reference and check for duplicate or board-revision-specific constraints.

8. Generate IP output products, compile the XPM library, then run elaboration, synthesis, implementation, and timing checks.

9. Program the card, attach the expected QSFP28 peer, and validate GT lock, AXI-Stream handshakes, ILA traces, and external packet captures.

The current ILA connection expects these probe widths:

probe0  1 bit    init clock
probe1  512 bits ERNIC/CRC TX data
probe2  4 bits   CMAC packet-state machine
probe3  1 bit    AXI-Stream ready
probe4  1 bit    TX last
probe5  1 bit    TX valid
probe6  1 bit    CMAC reference-clock output
probe7  1 bit    GT locked
probe8  1 bit    adapter valid
probe9  1 bit    adapter last

Simulation

The Git history records these milestones:

• 2025-04-02: CMAC and ERNIC modules running together in simulation;
• 2025-04-08: RDMA Send simulation completed and a simulation-oriented snapshot retained;
• 2025-04-08: first hardware-oriented integration did not produce data;
• 2025-04-20: a later hardware-deployment modification was committed.

These are development records, not independently reproducible benchmark claims. The repository does not contain the original Vivado project, simulator scripts, waveform database, packet capture, timing report, or bitstream.

For simulation, use the ernic/test/ stage as a reference and recreate the vendor IP simulation products in Vivado. Select exactly one version of each module. The checked-in snapshots require integration cleanup before they can be treated as a clean CI target.

Known limitations

• The repository is source/history oriented and is not a turnkey build.
• Vendor IP configuration and generated simulation/synthesis products are missing.
• Multiple source snapshots define identical module names.
• Some retained snapshots contain incomplete port edits or syntax mismatches. For example, the final parent connects tx_axis_tready to exdes_top, while the retained exdes_top port list does not expose that signal.
• The latest commit message records hardware deployment work, but no checked-in bitstream, ILA capture, packet capture, timing report, or throughput result can independently verify line-rate operation.
• Only the transmit-oriented board top is exposed; this is not a complete host-managed, production RDMA NIC stack.

Resume-ready summary

Integrated AMD/Xilinx ERNIC and CMAC UltraScale+ into a 512-bit AXI-Stream FPGA RDMA transmit prototype for Alveo U50, including AXI4-Lite/QP configuration, WQE-driven test traffic, iCRC, cross-clock buffering, board constraints, and ILA-based bring-up.

More detailed Chinese and English wording is available in docs/RESUME.md.

Attribution and licensing

This project contains AMD/Xilinx-generated example-design files with their own copyright notices and usage restrictions, plus project-specific integration and modifications by LLYniku. See NOTICE.md.

There is currently no repository-wide open-source license. Public visibility does not by itself grant permission to reuse every file. Review the notice in each vendor-derived file and the applicable AMD/Xilinx IP license before reuse or redistribution.