Building a Telco Test Lab Using srsRAN – #2 CPU Isolation

16. November 2025

This is the second blog post of a series called “Building a Telco Test Lab Using srsRAN.” In this chapter of the series, I will talk about CPU isolation.
For demonstration, I will use Ubuntu 24.04 with the real-time kernel installed. All steps should be applicable to other Linux distributions as well.

Planning the Deployment & CPU Core Separation

Before tuning, it’s worth deciding how to separate CPU cores. In this post, I will focus on single-socket systems with a single NUMA node.
Multi-socket or multi-NUMA setups require more complex affinity management, which I’ll cover in a future post.

Housekeeping vs. Isolated Cores

In order to reserve exclusive CPU cores for our real-time applications like the srsRAN gNB or DU, I separate the processes running on the host into two different groups: housekeeping cores and isolated cores.

• Housekeeping cores handle all system-level tasks and user-space services.
• Isolated cores are reserved for real-time workloads.

To save as many cores as possible for real-time applications, I also recommend removing or disabling everything that is not strictly required but could generate interrupts. On servers, I recommend removing or disabling all devices such as network adapters or other PCI devices that are not strictly needed for the deployment but could generate interrupts (IRQs). On regular workstation PCs, I recommend disabling things like Wi-Fi, audio, or Bluetooth adapters if available.

Disable systemd Services

Apart from removing unused hardware, it also helps to disable all on-board services that are not required on a real-time system. Ubuntu 24.04 ships with a lot of background daemons that add noise, wake up periodically, or trigger interrupts, all of which can affect deterministic performance. On a fresh installation, I usually disable the following services:

# Cloud-init
- cloud-init-local.service
- cloud-init.service
- cloud-config.service
- cloud-final.service

# First-boot and filesystem helpers
- blk-availability.service
- e2scrub_reap.service
- finalrd.service

# Desktop / user-land helpers
- apport.service
- gpu-manager.service
- ModemManager.service
- networkd-dispatcher.service
- open-iscsi.service
- open-vm-tools.service
- pollinate.service
- thermald.service
- ubuntu-advantage.service
- ua-reboot-cmds.service

# Sysadmin daemons that add periodic load
- multipathd.service
- sysstat.service
- unattended-upgrades.service
- ufw.service
- lvm2-monitor.service
- open-iscsi.service
- rsyslog.service
- systemd-networkd-wait-online.service

# Time sync services
- systemd-timesyncd.service
- ntpsec.service
- chrony.serivce

I recommend disabling them early in the setup, before starting any tuning or CPU isolation work. This keeps the system as quiet and predictable as possible. Make sure not to disable anything that you might need in the future.

Understanding Hyperthreading

Hyperthreading in real-time systems is a debated topic. With hyperthreading enabled, the system can deliver more performance, but on the other hand it becomes less deterministic. Based on my personal experience, hyperthreading can be used in real-time systems as long as the system is not pushed to its performance limits. Modern operating system schedulers are aware of the underlying architecture and ensure that one thread does not starve its sibling. However, this is not guaranteed.

In this blog post, I will configure the system with hyperthreading enabled. In my opinion, this configuration is the best choice when multiple applications need to run on the same machine, such as the gNB and the core network. In a future post, I will describe how to configure a system that is dedicated exclusively to the gNB. But I also want to mention here that this is not a production ready configuration, its for testing purposes. It gives flexibilty without the need to change anything.

With hyperthreading enabled, every physical core has a logical sibling that shares hardware resources such as caches and execution units. These pairs are visible as separate CPU IDs in Linux but correspond to the same physical core.

Both siblings of a hyperthreaded pair must always stay together, either both as housekeeping or both as isolated cores. If we were to split them into two different groups, a CPU core in the housekeeping group could steal resources from the isolated group.

You can see their relation in the output of lscpu:

$ lscpu -p
# CPU,Core,Socket,Node,,L1d,L1i,L2,L3
[...]
0,0,0,0,,0,0,0,0
[...]
12,0,0,0,,0,0,0,0
[...]

In this example, the CPU has 12 physical cores and 12 hyperthreading siblings. Logical CPUs 0 and 12 share the same Core (0), which means they belong to the same physical core. Both siblings share execution units, cache, and other hardware resources, which is why they must always be assigned to the same group.

In this example setup, I use:

• Housekeeping cores: 0,12
• Isolated cores: 1–11,13–23

Configuring Systemd CPU Affinity

To ensure that system services only use the housekeeping cores, edit the main systemd configuration file /etc/systemd/system.conf, uncomment, and set the CPUAffinity parameter as follows:

CPUAffinity=0,12

As a next step, I usually set the systemd user.slice and system.slice to the same value. Use the following commands to adjust both slices:

sudo systemctl set-property system.slice AllowedCPUs=0,12
sudo systemctl set-property user.slice AllowedCPUs=0,12

Check that everything was applied correctly using these commands:

sudo systemctl show system.slice --property AllowedCPUs
sudo systemctl show user.slice --property AllowedCPUs

To access those isolated cores, I use a cgroup. For this cgroup, I usually set the cpuset and memset. You can use the following script to create the cgroup. Make sure to replace the isolated cores, in this case, 1-11,13-23, with your isolated CPU core set.

sudo mkdir -p /sys/fs/cgroup/gnb/
echo 0 | sudo tee /sys/fs/cgroup/gnb/cpuset.mems
echo "1-11,13-23" | sudo tee /sys/fs/cgroup/gnb/cpuset.cpus

After the cgroup is created, use this command to write the PID of the shell into the cpuset.procs file, which enables the use of those isolated cores in this shell. For every new shell, you have to execute this command again.
The cgroup is not persistent after reboot.

echo $$ | sudo tee /sys/fs/cgroup/gnb/cgroup.procs

Adjust Kernel Arguments

The last remaining step is to adjust the RT kernel parameters. This step is very important and has to be carried out with precision. To apply the kernel arguments, edit the file /etc/default/grub and add the values under GRUB_CMDLINE_LINUX_DEFAULT.
For the start, I recommend using the following GRUB arguments:

nosoftlockup selinux=0 nmi_watchdog=0 crashkernel=auto softlockup_panic=0 audit=0 mce=off mitigations=off tsc=nowatchdog skew_tick=1 amd_iommu=on iommu=pt

After saving the file, run the following command to apply the settings to GRUB and reboot the machine. Make sure to execute the cgroup script again after reboot because the settings are not persistent.

sudo update-grub

Quick explanation for all parameters used:

Watchdogs / Lockup detectors / Debugging

• nosoftlockup – Disables the soft lockup detector. Reduces overhead but removes one safety check.
• nmi_watchdog=0 – Disables the NMI watchdog. Slight performance improvement but fewer safety checks.
• softlockup_panic=0 – Prevents kernel panic if a soft lockup is detected.
• audit=0 – Disables kernel audit logging, reducing overhead.

Security / Hardening

• selinux=0 – Disables SELinux.
• mitigations=off – Disables CPU vulnerability mitigations (Spectre, Meltdown, etc.) to improve performance at the cost of exposure to those exploits.

Error reporting / Reliability

• mce=off – Disables Machine Check Exception handling.
• crashkernel=auto – Reserves memory for crash dumps in case of a kernel panic.

Timing / TSC / Ticks

• tsc=nowatchdog – Disables the TSC clocksource watchdog, reducing overhead.
• skew_tick=1 – Staggers CPU tick timing to reduce latency spikes.

IOMMU / PCIe DMA

• amd_iommu=on – Enables AMD IOMMU for device isolation and DMA mapping.
• iommu=pt – Uses pass-through mode for lower latency while keeping VFIO compatibility.

I consider this set of arguments a “soft” CPU isolation because I am not moving any interrupts (IRQs). The IRQs on a system are distributed across all CPU cores for stability. If they don’t get enough CPU time, the system will become unstable and crash. In a future post, I will go into more detail on how to move IRQs, either using isolcpus or kernel parameters like kthread_cpus or irqaffinity as well as disabling C and P states.

Confirm the Setup Using Cyclictest

As a last step, I verify my setup using cyclictest. Cyclictest measures the difference between a thread’s intended wake-up time and the time at which it actually wakes up. I recommend doing this evaluation over 24 hours, and the expected maximum delay should be around 25 µs after tuning.

I run cyclictest in combination with stress-ng to simulate load on the system while measuring the wake-up time. I run stress-ng with an RT priority of 80 and cyclictest with an RT priority of 98. This mirrors more or less the priorities of the srsRAN gNB.
Use the following commands to run the stress test:

sudo nice -80 stress-ng --cpu 22 --io 22 --vm 22 --timeout 24h
sudo cyclictest --mlockall --priority 98 --distance 0 --threads 22 --duration 24h

You should see an output similar to this one:

# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 30.01 14.42 7.61 22/950 2520178
T: 0 (2519907) P:98 I:35 C: 882328 Min: 2 Act: 3 Avg: 2 Max: 20
T: 1 (2519908) P:98 I:35 C: 882228 Min: 1 Act: 2 Avg: 3 Max: 19
T: 2 (2519909) P:98 I:35 C: 881532 Min: 1 Act: 3 Avg: 3 Max: 23
T: 3 (2519910) P:98 I:35 C: 881993 Min: 2 Act: 3 Avg: 3 Max: 24
T: 4 (2519911) P:98 I:35 C: 881774 Min: 2 Act: 3 Avg: 3 Max: 21
...

In case you see values above 25µs you have to debug your grub kernel arguments. If the values are around 25µs or below, the system is optimized for real-time workloads, and you can continue with the next chapter.

Make all settings persistent

In order to make all settings of this chapter persist after reboot I add them to the TuneD startup script and profile. An updated version of both, tuned.conf and startup.sh, can be found here.