forks-thunderbolt

AppMana fork of the Linux kernel Thunderbolt drivers and our out-of-tree thunderbolt_ibverbs (usb4_rdma RDMA) transport, in one repo so the three host-to-host Thunderbolt stacks share a single negotiation header instead of hand-synced copies:

thunderbolt — the core XDomain/USB4 driver.
thunderbolt_net — IP-over-Thunderbolt (tbnet).
thunderbolt_ibverbs — the usb4_rdma RDMA transport + userspace provider.
rdma_rxe — AppMana's soft-RoCE fallback override for rxe_lan.

All three negotiate the same XDomain connection and shared the same soft-reconnect bug; the fix lives once in drivers/thunderbolt/thunderbolt_negotiation.h (generation gate + handshake re-arm + the TB_XNEG_* installable macros), with KUnit, run via drivers/thunderbolt_ibverbs/tools/run-kunit.sh (kunit.py on an overlaid v6.17 tree, x86_64 qemu; the fleet/host kernels lack CONFIG_KUNIT).

upstream  = git://git.kernel.org/pub/scm/linux/kernel/git/westeri/thunderbolt.git
origin    = git@github.com:AppMana/forks-thunderbolt.git
branch    = pub/tbfix-v6.17    (deployed on the AppMana fleet)
base      = v6.17 + 3 backports already in linux-hwe-6.17

What's here

drivers/thunderbolt/            full subsystem source (patched) + the shared
                                thunderbolt_negotiation.h (single source of truth)
drivers/net/thunderbolt/        tbnet driver source (patched)
drivers/infiniband/sw/rxe/      RXE source from Ubuntu HWE 6.17 with AppMana
                                `rdma link del` protection for `rxe_lan`
drivers/thunderbolt_ibverbs/    usb4_rdma RDMA driver, userspace provider, dkms,
                                tests (merged from the old thunderbolt-ibverbs repo)
dkms/                           DKMS scaffolding (dkms.conf, Makefile)
packaging/debian/           Debian package metadata for thunderbolt-tbfix-dkms
scripts/
  oot-build.sh              fast iteration: stage at ~/src/tb-oot, build, hot-swap
  export-dkms-payload.sh    write byte-identical DKMS bundle into the appmana repo
tools/ci/
  distro-package.sh         build thunderbolt-tbfix-dkms_<version>_all.deb
  distro-install.sh         verify the .deb can build through DKMS
tests/
  run-smoke.sh              60-s NCCL hostnet sweep on the 3-node chain
  run-durability.sh         192 GiB allreduce reproducer (~30 min, the wedge gate)

How to use

Read docs/thunderbolt_fix.md in the parent appmana repo. It covers:

Building OOT for fast iteration (scripts/oot-build.sh --swap).
Building the DKMS payload for fleet deploy (scripts/export-dkms-payload.sh).
Running the smoke + durability tests.
Deploying via Ansible (playbook_worker.yaml).
Reverting to in-tree drivers.
Rebasing on a newer kernel when the fleet bumps.
Preparing upstream topic patches for manual review.

Who owns the link: `usb4_rdma` vs `thunderbolt_net`

Both drivers can be loaded at once, and both advertise their XDomain services (tbverbs -> thunderbolt_ibverbs, network -> thunderbolt_net). But one driver owns the link's DMA data path at a time, and which one is a runtime switch — no rmmod, no reboot:

# hand the link to thunderbolt_net (IP over Thunderbolt: tb-ch* netdevs)
echo tbnet > /sys/module/thunderbolt_ibverbs/parameters/link_owner

# hand it back to usb4_rdma (RDMA: usb4_rdma* ib_devices)  [default]
echo rdma  > /sys/module/thunderbolt_ibverbs/parameters/link_owner

# who owns it right now
grep link_owner /sys/kernel/debug/thunderbolt_ibverbs/summary

rdma is the default; set it at boot with options thunderbolt_ibverbs link_owner=tbnet in /etc/modprobe.d/.

link_owner=tbnet — thunderbolt_ibverbs unpublishes its usb4_rdma ib_devices and disables its DMA tunnels. It does not tear down rings or negotiation, so the switch back is cheap. thunderbolt_net then establishes its tunnel and the tb-ch* netdevs start passing packets.
link_owner=rdma — each rail re-runs its HELLO negotiation, re-enables its tunnel and republishes the ib_device. ib_send_lat works again.

The switch is idempotent and applied on a bounded workqueue (no RTNL, no hotplug notifiers, no ICM interaction), so it cannot wedge the chain the way a module reload does.

Why `thunderbolt_net` used to be a zombie

Before tbfix 2.13 / tbv 0.2.36, thunderbolt_net on a chain node looked healthy — netdev up, carrier=1, correct IPs, routes resolving — and passed zero packets. That was not a resource conflict with usb4_rdma: the two use separate NHI rings and separate XDomain hopids, and both hop entries are programmed. Reading the hop entries back off the routers showed the real mechanism: tbnet's inbound lane-adapter entries were deactivated (enable=0) while tbv's entries on the same adapters were enable=1 and carrying traffic.

tbnet got stuck that way because its ThunderboltIP session is one-shot: login_work and connected_work both early-return once carrier is on, so once both ends latch "login complete" nothing ever re-LOGINs — and this fleet demonstrably loses the hotplug edges upstream relies on to tear the session down (a neighbour rebooting can produce no thunderbolt event on the surviving node). thunderbolt_ibverbs survives the identical events because its HELLO negotiation is level-triggered.

The fix (in thunderbolt_net, using the stock spec LOGIN — the negotiation is not reimplemented) is a level-triggered session verify: while carrier is on, re-check that the DMA tunnel behind the session is still alive (tb_xdomain_paths_active() reads every hop entry back from the routers), and if it died, tear the session down and re-run a normal LOGIN.

# how often to re-check the tunnel behind an established session (ms); 0 disables
cat /sys/module/thunderbolt_net/parameters/session_verify_ms   # default 5000

A healthy link logs nothing; recovery logs ThunderboltIP session lost its DMA tunnel; tearing down and re-logging in.

Install on Ubuntu

From the AppMana apt repository

Once AppMana/apt has published the release, install with:

curl -fsSL https://appmana.github.io/apt/appmana-archive-keyring.gpg \
  | sudo gpg --dearmor -o /usr/share/keyrings/appmana-archive-keyring.gpg

echo 'deb [arch=amd64 signed-by=/usr/share/keyrings/appmana-archive-keyring.gpg] https://appmana.github.io/apt noble main' \
  | sudo tee /etc/apt/sources.list.d/appmana.list

sudo apt update
sudo apt install thunderbolt-tbfix-dkms

From GitHub Releases

Download thunderbolt-tbfix-dkms_<version>_all.deb from:

https://github.com/AppMana/thunderbolt-tbfix/releases

Then install it:

sudo apt install build-essential dkms kmod make "linux-headers-$(uname -r)"
sudo apt install ./thunderbolt-tbfix-dkms_<version>_all.deb
dkms status -m thunderbolt-tbfix

The package stages source under /usr/src/thunderbolt-tbfix-<version> and runs dkms autoinstall. It intentionally does not reload thunderbolt or thunderbolt_net; fleet reload ordering remains owned by Ansible or an explicit maintenance command.

The split-package build also emits rdma-rxe-appmana_<version>_all.deb. That package installs an override rdma_rxe.ko which does not advertise IBK_ALLOW_USER_UNREG. On AppMana, rxe_lan is a live NCCL fallback rail; rdma link del rxe_lan should fail fast instead of unregistering the device while Ray/NCCL still owns QPs.

Verify the installed module path:

modinfo thunderbolt | sed -n '1,8p'
modinfo thunderbolt_net | sed -n '1,8p'

filename should point under /lib/modules/<kernel>/updates/, not the stock /kernel/drivers/... tree.

Build a Debian package

Build the DKMS .deb locally:

tools/ci/distro-package.sh ubuntu

The artifact is written to dist/thunderbolt-tbfix-dkms_<version>_all.deb. Tags matching v* publish the .deb and its .sha256 file to GitHub Releases in AppMana/thunderbolt-tbfix. The public apt repository in AppMana/apt consumes those release assets.

Development process

Install build and test dependencies on an Ubuntu development host:

sudo apt update
sudo apt install build-essential dkms git kmod "linux-headers-$(uname -r)" \
  linux-tools-common linux-tools-generic trace-cmd

Fast one-host edit/build loop:

scripts/oot-build.sh
scripts/oot-build.sh --install

--install writes the freshly built modules to /lib/modules/$(uname -r)/updates/ and runs depmod. To live-swap on a test host:

scripts/oot-build.sh --swap

Do not run --swap on a production chain node unless the Thunderbolt link can be interrupted. It unloads and reloads thunderbolt_net and thunderbolt.

DKMS/package loop:

tools/ci/distro-package.sh ubuntu
sudo apt install ./dist/thunderbolt-tbfix-dkms_<version>_all.deb
sudo dkms build -m thunderbolt-tbfix -v <version> -k "$(uname -r)" --force
sudo dkms install -m thunderbolt-tbfix -v <version> -k "$(uname -r)" --force

Container verification, without touching host modules:

docker run --rm -v "$PWD:/work" -w /work ubuntu:24.04 \
  bash tools/ci/distro-package.sh ubuntu

docker run --rm -v "$PWD:/work" -w /work ubuntu:24.04 \
  bash tools/ci/distro-install.sh 'dist/thunderbolt-tbfix-dkms_*.deb'

The container install check verifies the .deb metadata and /usr/src staging. It does not compile the module by default because this tbfix branch is for the fleet's 6.17 Thunderbolt tree, while stock Ubuntu 24.04 containers install 6.8 headers. To compile in a matching kernel-header environment:

TBFIX_VERIFY_DKMS_BUILD=1 tools/ci/distro-install.sh \
  'dist/thunderbolt-tbfix-dkms_*.deb'

Functional tests:

tests/run-smoke.sh
tests/run-durability.sh

run-smoke.sh is the quick gate. run-durability.sh is the wedge gate and should complete the 192 GiB allreduce target before fleet rollout.

Tracing and diagnostics:

sudo mount -t tracefs nodev /sys/kernel/tracing 2>/dev/null || true
sudo trace-cmd list | grep -Ei 'thunderbolt|tbnet|nhi|irq'
sudo trace-cmd record -e thunderbolt:* -e napi:* -e irq:* -- sleep 30
sudo trace-cmd report | less

Useful live checks:

sudo dmesg -T | grep -Ei 'thunderbolt|tb-ch|DMA paths|login|host found' | tail -100
ls -l /sys/bus/thunderbolt/devices
for d in /sys/bus/thunderbolt/devices/*; do
  [ -e "$d" ] || continue
  echo "== $d =="
  for f in device_name unique_id rx_speed tx_speed rx_lanes tx_lanes authorized; do
    [ -r "$d/$f" ] && echo "$f=$(cat "$d/$f")"
  done
done
ip -br link | grep -E 'tb-ch|thunderbolt'

For NHI interrupt-mask debugging, inspect Thunderbolt debugfs if available:

sudo find /sys/kernel/debug/thunderbolt -maxdepth 3 -type f -print 2>/dev/null

Capture these before and after smoke/durability runs when changing drivers/thunderbolt/nhi.c, drivers/thunderbolt/path.c, or drivers/net/thunderbolt/main.c.

Branches

pub/tbfix-v6.17 — deployed DKMS branch. Includes the Ubuntu HWE backport alignment, Thunderbolt networking/ring reliability work, DKMS packaging, and ICM hotplug diagnostics.
pub/tbfix-v6.17-hotplug — storage-hotplug split branch. Carries only the drivers/thunderbolt/icm.c hotplug work on top of the DKMS packaging point; no drivers/net/thunderbolt changes.
master — local mirror of upstream/master (Mika Westerberg's tree).

Do not submit the fleet branch upstream. For upstream work, create a fresh topic branch from upstream/master and apply only the minimal subsystem-specific change. Thunderbolt networking changes touch netdev; storage hotplug changes should not.

Operational caveats

An ICM wedge is cleared only by removing power. Symptom: thunderbolt 0000:xx:00.0: failed to send driver ready to ICM, probe ... error -110, and no usb4_rdma* device. Measured 2026-07-09 on Maple Ridge: warm reboot, PCI secondary-bus reset, D3cold via runtime PM (no _PR3 on these boards), and the WMI force_power toggle all fail. systemctl poweroff to S5 followed by power-on clears it every time. Arm Wake-on-LAN first (ethtool -s <if> wol g) or the node needs a physical power button press — some boards (appmana-002, appmana-020) ignore WoL from S5 regardless.
Never live-reload the modules on a chain node. rmmod/modprobe of thunderbolt_ibverbs under traffic wedges the ICM. The deploy path is dpkg -i then rmmod the module before rebooting (a warm reboot with active NHI rings leaves the ICM mid-transaction) — see appmana-management/.../scripts/tb-chain-reboot-cover.sh, which drains, unloads, reboots a pairwise non-adjacent set, and verifies negotiation.
A rail's peers only re-negotiate after its neighbour also rolls. After rebooting one half of the chain, data-ready-rails=0 on those nodes is expected until their neighbours reboot too; do not chase it.
Loss recovery is deadline-armed since 0.2.25. Before that, every lost frame recovered on a ~100 ms grid regardless of tbv_retransmit_base_ms, because the backoff deadline was only a threshold checked when the QP timeout work happened to run, and every arming site scheduled that work at min3(ack_timeout, 1000ms, TBV_READ_RESP_RETRY_MS=100ms). Symptom to watch for in /sys/kernel/debug/thunderbolt_ibverbs/summary: data_rx_ack_match_over_64ms climbing with data_wr_retransmit, and data_rx_duplicate_ack close to the retransmit count (spurious resends).
data_rx_bad_header is not necessarily wire corruption. Until 0.2.26 the RX path never inspected frame->flags, so frames the NHI had already marked RING_DESC_CRC_ERROR / RING_DESC_BUFFER_OVERRUN were parsed anyway. The data_rx_crc_error / data_rx_overrun counters now separate hardware-flagged corruption from software framing bugs. Note enum ring_desc_flags aliases TX and RX meanings on the same bits (0x1 ISOCH/CRC_ERROR, 0x4 POSTED/ BUFFER_OVERRUN) — the check is only valid on an RX completion.
Hardware E2E flow control is NOT a throughput win. The c5ec614 commit message reports ~28 Gb/s duplex with E2E vs ~3 Gb/s on software credits; that came from a synthetic bandwidth sweep and does not reproduce on real NCCL message shapes, where the two are about the same. E2E remains off for the native backend: it buys nothing measurable and carries the Maple Ridge ring-teardown lockup and AMD TX-completion wedge history. Do not re-derive a policy change from that commit message alone.

Why patches not a submodule

forks-* siblings of the appmana monorepo are independent git repos by convention (see forks-sglang, forks-vllm-ampere). Ansible deploys from a flat in-repo copy of the DKMS payload at appmana-management/src/appmana_management/files/thunderbolt_net/tbfix-dkms/.

The scripts/export-dkms-payload.sh script in this fork rewrites that copy from HEAD. If the fork advances, run the script and commit the appmana-side change; diff -r between fork export and appmana copy must be empty before deploy.

License

Linux kernel sources are GPL-2.0-only; the DKMS/scripts overlay matches.