This is the 3rd chapter in my blog series “Building a Telco Test Lab Using srsRAN”. In this chapter, I show how I build and install DPDK from source, install the necessary drivers, create a Virtual Function, and bind it to vfio-pci.
DPDK is a set of libraries and drivers that allows applications to process network packets entirely in user space, bypassing the Linux kernel’s networking stack. This gives massively higher throughput and lower latency because packets are moved directly between the NIC and the application using poll-mode drivers.
Enable SR-IOV and IOMMU
As a first step, I ensure that SR-IOV and IOMMU are supported and enabled in the BIOS. This depends on the hardware, so consult the vendor documentation to see how to enable both. To verify that they are enabled on Ubuntu 24.04, I use the following commands.
IOMMU
$ sudo dmesg | grep -e IOMMU -e DMAR
[ 0.012923] ACPI: DMAR 0x000000006F55A000 0000D8 (v01 DELL PE_SC3 00000001 DELL 00000001)
[ 0.012942] ACPI: Reserving DMAR table memory at [mem 0x6f55a000-0x6f55a0d7]
[ 0.144955] DMAR: IOMMU enabled
...
SR-IOV
$ lspci | grep -i ethernet
01:00.0 Ethernet controller: Intel Corporation Ethernet Controller E810-C for SFP (rev 02)
01:00.1 Ethernet controller: Intel Corporation Ethernet Controller E810-C for SFP (rev 02)
01:00.2 Ethernet controller: Intel Corporation Ethernet Controller E810-C for SFP (rev 02)
01:00.3 Ethernet controller: Intel Corporation Ethernet Controller E810-C for SFP (rev 02)
$ sudo lspci -s 01:00.0 -vvv | grep SR-IOV
Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
Build DPDK
Install the following dependencies:
sudo apt install build-essential tar wget python3-pip pipx libnuma-dev python3-pyelftools unzip git meson ninja-build
Download and build DPDK 24.11.2 using the following instructions. Be aware that this will install DPDK into the system path. If you need a different setup, refer to the official documentation.
wget https://fast.dpdk.org/rel/dpdk-24.11.2.tar.xz -P /tmp/
tar xvf /tmp/dpdk-24.11.2.tar.xz -C ~/
mv ~/dpdk-*24.11.2* ~/dpdk-24.11.2
cd ~/dpdk-24.11.2/
meson setup build
cd build
ninja
sudo meson install
sudo ldconfig
Choosing the Driver
DPDK supports multiple drivers. These are for example:
- vfio_pci
- igb_uio
- uio_pci_generic
vfio_pci offers the best performance and security, which is why I usually use it. If IOMMU is not available, vfio_pci cannot be used, in that case I recommend igb_uio. igb_uio provides similar performance but lacks the security features of vfio_pci. In the following, I will show how to use both.
vfio-pci
This driver is included in Ubuntu’s kernel modules.
Check if it is loaded:
lsmod | grep vfio_pci
If not loaded, insert it:
sudo modprobe vfio_pci
lsmod | grep vfio_pci
If the module does not load, check the kernel logs (dmesg). I recommend adding vfio_pci to /etc/modules so it loads on boot.
igb_uio
This driver is not shipped with Ubuntu, so we must build it:
git clone http://dpdk.org/git/dpdk-kmods
cd dpdk-kmods/linux/igb_uio/
make
sudo make install
Before loading igb_uio, ensure that uio is loaded:
lsmod | grep uio
sudo modprobe uio
sudo insmod igb_uio.ko
Check dmesg for errors if something fails. Add igb_uio to /etc/modules if you want it to load automatically on boot.
Creating a Virtual Function Using SR-IOV
When binding a DPDK driver to a network interface, the interface disappears from the kernel. This means tools like ip or ifconfig cannot configure it anymore, and PTP will not work on it.
To allow both PTP and OFH traffic, I create a Virtual Function (VF) so that multiple services can share the same physical interface. To create a single VF for interface enp81s0f0np0 run this command:
echo 1 | sudo tee /sys/class/net/enp81s0f0np0/device/sriov_numvfs
VFs are initialized with default settings. To set a MAC and disable spoofcheck, run the following command (don’t forget to adjust the network interface name). Generate a radom but unique MAC for the VF.
sudo ip link set enp81s0f0np0 vf 0 mac 00:33:22:33:00:11 spoofchk off
Some drivers (e.g., ixgbvf) require a reload after setting the MAC to apply it:
sudo modprobe -r ixgbvf
sudo modprobe ixgbvf
Bind Interface to Driver
After loading the necessary driver, I check DPDK’s interface status:
dpdk-devbind.py -s
Network devices using DPDK-compatible driver
============================================
Network devices using kernel driver
===================================
0000:01:00.0 'Ethernet Controller E810-C for SFP 1593' if=enp1s0f0np0 drv=ice unused=vfio-pci *Active*
0000:01:01.0 'Ethernet Adaptive Virtual Function 1889' drv=iavf unused=vfio-pci
0000:01:00.1 'Ethernet Controller E810-C for SFP 1593' if=enp1s0f1np1 drv=ice unused=vfio-pci
0000:01:00.2 'Ethernet Controller E810-C for SFP 1593' if=enp1s0f2np2 drv=ice unused=vfio-pci
0000:01:00.3 'Ethernet Controller E810-C for SFP 1593' if=enp1s0f3np3 drv=ice unused=vfio-pci
0000:07:00.0 'RTL8125 2.5GbE Controller 8125' if=enp7s0 drv=r8169 unused=vfio-pci *Active*
If everything looks correct, bind the VF:
sudo dpdk-devbind.py --bind=vfio-pci 0000:01:01.0
Verify the binding:
dpdk-devbind.py -s
Network devices using DPDK-compatible driver
============================================
0000:01:01.0 'Ethernet Adaptive Virtual Function 1889' drv=vfio-pci unused=iavf
Network devices using kernel driver
===================================
0000:01:00.0 'Ethernet Controller E810-C for SFP 1593' if=enp1s0f0np0 drv=ice unused=vfio-pci *Active*
...
Verify That DPDK and the Interface Are Working
I typically run testpmd to verify that the DPDK installation is correct. Make sure you have bound at least one interface to a DPDK compatible driver:
sudo dpdk-testpmd -l 0-3 -n 4 -- --port-topology=chained --forward-mode=macswap
Make Settings Persistent
To make the setup persistent, I add the necessary commands to the startup script. An updated version of the script will be available here.