Building a Telco Test Lab Using srsRAN – #3 DPDK

This is the 3rd chapter in my blog series “Building a Telco Test Lab Using srsRAN”. In this chapter, I show how I build and install DPDK from source, install the necessary drivers, create a Virtual Function, and bind it to vfio-pci.

DPDK is a set of libraries and drivers that allows applications to process network packets entirely in user space, bypassing the Linux kernel’s networking stack. This gives massively higher throughput and lower latency because packets are moved directly between the NIC and the application using poll-mode drivers.

Enable SR-IOV and IOMMU

As a first step, I ensure that SR-IOV and IOMMU are supported and enabled in the BIOS. This depends on the hardware, so consult the vendor documentation to see how to enable both. To verify that they are enabled on Ubuntu 24.04, I use the following commands.

IOMMU

$ sudo dmesg | grep -e IOMMU -e DMAR

[    0.012923] ACPI: DMAR 0x000000006F55A000 0000D8 (v01 DELL   PE_SC3   00000001 DELL 00000001)
[    0.012942] ACPI: Reserving DMAR table memory at [mem 0x6f55a000-0x6f55a0d7]
[    0.144955] DMAR: IOMMU enabled
...

SR-IOV

$ lspci | grep -i ethernet

01:00.0 Ethernet controller: Intel Corporation Ethernet Controller E810-C for SFP (rev 02)
01:00.1 Ethernet controller: Intel Corporation Ethernet Controller E810-C for SFP (rev 02)
01:00.2 Ethernet controller: Intel Corporation Ethernet Controller E810-C for SFP (rev 02)
01:00.3 Ethernet controller: Intel Corporation Ethernet Controller E810-C for SFP (rev 02)

$ sudo lspci -s 01:00.0 -vvv | grep SR-IOV

Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)

Build DPDK

Install the following dependencies:

sudo apt install build-essential tar wget python3-pip pipx libnuma-dev python3-pyelftools unzip git meson ninja-build

Download and build DPDK 24.11.2 using the following instructions. Be aware that this will install DPDK into the system path. If you need a different setup, refer to the official documentation.

wget https://fast.dpdk.org/rel/dpdk-24.11.2.tar.xz -P /tmp/
tar xvf /tmp/dpdk-24.11.2.tar.xz -C ~/
mv ~/dpdk-*24.11.2* ~/dpdk-24.11.2
cd ~/dpdk-24.11.2/
meson setup build
cd build
ninja
sudo meson install
sudo ldconfig

Choosing the Driver

DPDK supports multiple drivers. These are for example:

  • vfio_pci
  • igb_uio
  • uio_pci_generic

vfio_pci offers the best performance and security, which is why I usually use it. If IOMMU is not available, vfio_pci cannot be used, in that case I recommend igb_uio. igb_uio provides similar performance but lacks the security features of vfio_pci. In the following, I will show how to use both.

vfio-pci

This driver is included in Ubuntu’s kernel modules.

Check if it is loaded:

lsmod | grep vfio_pci

If not loaded, insert it:

sudo modprobe vfio_pci
lsmod | grep vfio_pci

If the module does not load, check the kernel logs (dmesg). I recommend adding vfio_pci to /etc/modules so it loads on boot.

igb_uio

This driver is not shipped with Ubuntu, so we must build it:

git clone http://dpdk.org/git/dpdk-kmods
cd dpdk-kmods/linux/igb_uio/
make
sudo make install

Before loading igb_uio, ensure that uio is loaded:

lsmod | grep uio
sudo modprobe uio
sudo insmod igb_uio.ko

Check dmesg for errors if something fails. Add igb_uio to /etc/modules if you want it to load automatically on boot.

Creating a Virtual Function Using SR-IOV

When binding a DPDK driver to a network interface, the interface disappears from the kernel. This means tools like ip or ifconfig cannot configure it anymore, and PTP will not work on it.

To allow both PTP and OFH traffic, I create a Virtual Function (VF) so that multiple services can share the same physical interface. To create a single VF for interface enp81s0f0np0 run this command:

echo 1 | sudo tee /sys/class/net/enp81s0f0np0/device/sriov_numvfs

VFs are initialized with default settings. To set a MAC and disable spoofcheck, run the following command (don’t forget to adjust the network interface name). Generate a radom but unique MAC for the VF.

sudo ip link set enp81s0f0np0 vf 0 mac 00:33:22:33:00:11 spoofchk off

Some drivers (e.g., ixgbvf) require a reload after setting the MAC to apply it:

sudo modprobe -r ixgbvf
sudo modprobe ixgbvf

Bind Interface to Driver

After loading the necessary driver, I check DPDK’s interface status:

dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================

Network devices using kernel driver
===================================
0000:01:00.0 'Ethernet Controller E810-C for SFP 1593' if=enp1s0f0np0 drv=ice unused=vfio-pci *Active*
0000:01:01.0 'Ethernet Adaptive Virtual Function 1889' drv=iavf unused=vfio-pci
0000:01:00.1 'Ethernet Controller E810-C for SFP 1593' if=enp1s0f1np1 drv=ice unused=vfio-pci
0000:01:00.2 'Ethernet Controller E810-C for SFP 1593' if=enp1s0f2np2 drv=ice unused=vfio-pci
0000:01:00.3 'Ethernet Controller E810-C for SFP 1593' if=enp1s0f3np3 drv=ice unused=vfio-pci
0000:07:00.0 'RTL8125 2.5GbE Controller 8125' if=enp7s0 drv=r8169 unused=vfio-pci *Active*

If everything looks correct, bind the VF:

sudo dpdk-devbind.py --bind=vfio-pci 0000:01:01.0

Verify the binding:

dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:01:01.0 'Ethernet Adaptive Virtual Function 1889' drv=vfio-pci unused=iavf

Network devices using kernel driver
===================================
0000:01:00.0 'Ethernet Controller E810-C for SFP 1593' if=enp1s0f0np0 drv=ice unused=vfio-pci *Active*
...

Verify That DPDK and the Interface Are Working

I typically run testpmd to verify that the DPDK installation is correct. Make sure you have bound at least one interface to a DPDK compatible driver:

sudo dpdk-testpmd -l 0-3 -n 4 -- --port-topology=chained --forward-mode=macswap

Make Settings Persistent

To make the setup persistent, I add the necessary commands to the startup script. An updated version of the script will be available here.

Comments
Join the Discussion and Share Your Opinion
Add a Comment

Schreibe einen Kommentar

Deine E-Mail-Adresse wird nicht veröffentlicht. Erforderliche Felder sind mit * markiert