---

Introduction

On February 2024, a dangerous deadlock vulnerability was identified in the Linux kernel component dealing with Mellanox (NVIDIA) network drivers—specifically, in the way it disables Accelerated Receive Flow Steering (aRFS) in the mlx5e driver. This issue, tracked as CVE-2024-27014, can lead to complete system hangs or service disruption—critical for servers and environments using high-performance Mellanox adapters (commonly found in data centers and cloud infrastructure).

Let’s break down what happened, why it’s dangerous, how it works, and how the kernel patch neutralizes the bug.

What is the Vulnerability?

When the mlx5e driver disables aRFS (Accelerated Receive Flow Steering), it must clean up any pending aRFS tasks (“works”). It does this while holding a mutex called priv->state_lock to avoid race conditions. Unfortunately, cancelling these works synchronously (via cancel_work_sync()) can wait for a work handler that's itself trying to acquire the same state_lock—causing a circular locking dependency — also known as a deadlock.

Disable aRFS: Grab a lock (state_lock), then clean up aRFS stuff.

- aRFS Cleanup: May have already scheduled cleanups (works), so it waits for them to finish before truly releasing resources.

But: Those works, when running, also want to grab the same state_lock.

- Result: A classic deadlock: one thread holds the lock waiting for another thread to finish—a thread that needs the same lock.

Real-world log output captures the deadlock

WARNING: possible circular locking dependency detected
...
lock(&priv->state_lock);
                           lock((work_completion)(&rule->arfs_work));
                           lock(&priv->state_lock);
lock((work_completion)(&rule->arfs_work));

* DEADLOCK *

Kernel diagnostic tools (lockdep) flag this dangerous chain clearly.

Here’s a sketch of the critical operations in the vulnerable code

// Disabling aRFS from ethtool:
mutex_lock(&priv->state_lock);

cancel_work_sync(&arfs_rule->arfs_work);
// ^ Waits for arfs_work to finish, but arfs_work tries to grab state_lock

mutex_unlock(&priv->state_lock);

Meanwhile, the work handler (arfs_handle_work) also does

arfs_handle_work()
{
    mutex_lock(&priv->state_lock);  // DEADLOCK! if already held by the disabling thread
    // ... clean up rules ...
    mutex_unlock(&priv->state_lock);
}

How an Attacker Could Exploit (Exploit Details)

Because this is not a classic remote code execution or privilege escalation, but a denial of service (DoS), attackers with the ability to manage network settings or run privileged utilities (like ethtool) can:

PoC (Proof of Concept) Script Sketch

#!/bin/bash
DEV=eth
while true; do
    # Start network load (simulating high traffic)
    # Run in background - e.g. using iperf3 or similar

    # At the same time, trigger aRFS relevant settings
    ethtool -L $DEV combined 4  # set channels
    ethtool -L $DEV combined 2
    # Or toggle aRFS directly if supported

    sleep .1
done

Note: For root/privileged users only. On vulnerable kernels, eventually you’ll see a system hang or kernel warnings.

The Patch (How It Was Fixed)

Key idea:
The patch introduces an explicit aRFS state variable. Before adding any aRFS rules, it checks this state; if aRFS is disabled, no new rules are added, and the work handler responsible for cleanup avoids acquiring the deadlock-prone lock.

Patch Snippet

// Add state variable
priv->arfs_enabled = false;  // when disabling

// Before adding/removing rules:
if (!priv->arfs_enabled)
    return; // do nothing

// Work handler is updated:
arfs_handle_work()
{
    if (!priv->arfs_enabled)
        return;

    mutex_lock(&priv->state_lock);
    // cleanup
    mutex_unlock(&priv->state_lock);
}

Result: Scheduled works can notice aRFS is now “off” and do nothing (avoiding lock contention). This breaks the deadlock path and makes disabling aRFS robust.

Official Patch / Discussion:
- net/mlx5e: Prevent deadlock while disabling aRFS (lore.kernel.org)
- kernel.org commit link

Anyone using the Linux kernel v6.7 and recent with Mellanox ConnectX (mlx5) series adapters

- Especially in environments with heavy channel reconfiguration, dynamic aRFS changes, or aggressive networking tests

How to Mitigate

- Upgrade your kernel to a version including the above patch (mainline v6.7+ after early December 2023)
- If you must use an older kernel, avoid toggling aRFS or rapidly changing channel counts while high traffic is present.

Conclusion

CVE-2024-27014 is a classic example of synchronization gone wrong—a reminder that “safe” locking must always consider less-obvious paths like workqueues and events. For operators running high-speed Mellanox cards on Linux, make sure you’re patched and avoid unnecessary aRFS/channel toggling unless you know your kernel includes the fix.

For the latest details, see the original kernel patch:
👉 https://lore.kernel.org/netdev/20231214160748.9844-1-leon@kernel.org/

Timeline

Published on: 05/01/2024 06:15:20 UTC
Last modified on: 06/17/2024 17:46:06 UTC