Back to Blog

Xid 79 GPU Has Fallen Off the Bus on Linux and How to Find the Cause

Christopher 4 min read
referencelinux

Xid 79 is an error code the NVIDIA driver writes to the Linux kernel log. NVIDIA's Xid catalog lists it as ROBUST_CHANNEL_GPU_HAS_FALLEN_OFF_THE_BUS with the description "GPU has fallen off the bus". Each Xid line has "NVRM: Xid" and the GPU's PCI address, with the code after a colon. NVIDIA's example line for Xid 79 looks like this.

NVRM: Xid (PCI:0000:5a:00): 79, GPU has fallen off the bus.

The driver can also log Xid 154 with it. Xid 154 gives the recovery action the GPU needs after the other Xid, such as "Node Reboot Required".

What it means

The driver tried to reach the GPU over its PCI Express connection and found it inaccessible. NVIDIA says hardware failures on the PCI Express link that bring the link down are a frequent cause. Failing GPU hardware and other driver issues can also cause it. NVIDIA's immediate action for Xid 79 is to restart the machine. Its investigatory action is to contact your support organization.

How to find the cause

Find the Xid lines and any PCI Express errors for the boot that failed. The -k option keeps only kernel messages. The -b -1 option selects the boot before the current one.

journalctl -k -b -1 | grep -E 'NVRM: Xid|PCIe Bus Error'

The kernel's Advanced Error Reporting (AER) driver logs PCI Express errors as "PCIe Bus Error" lines with a severity and a type. NVIDIA names the kernel PCI event logs as the place to look for the source of the link failure. When a "PCIe Bus Error" line for the GPU's address comes just before the Xid 79 line, it's likely that the link failed first.

If only the current boot appears in journalctl --list-boots, the journal isn't kept across reboots. Set Storage=persistent in /etc/systemd/journald.conf so journal data is stored under /var/log/journal, then wait for the next occurrence.

Check whether the driver powers the GPU down when it's idle. Runtime D3 power management needs a Turing or newer GPU. The driver turns it on by default only on supported Ampere or newer notebooks.

grep 'Runtime D3 status' /proc/driver/nvidia/gpus/*/power

Check whether Active State Power Management (ASPM) is on for the GPU's link. Replace 5a:00 with the bus and device numbers from your Xid line. "ASPM Disabled" on the "LnkCtl" line means the link doesn't use ASPM.

sudo lspci -vvv -s 5a:00 | grep 'LnkCtl:'

Fixes

Restart the machine to bring the GPU back, which is NVIDIA's immediate action for Xid 79.

When "Runtime D3 status" reads "Enabled", turn runtime D3 off as a test. Add this line to /etc/modprobe.d/nvidia.conf and reboot.

options nvidia "NVreg_DynamicPowerManagement=0x00"

When "PCIe Bus Error" lines come before the Xid or ASPM is enabled on the GPU's link, test with ASPM off by adding pcie_aspm=off to the kernel command line. On Fedora and RHEL, add it to every boot entry with grubby and reboot.

sudo grubby --update-kernel=ALL --args="pcie_aspm=off"

If Xid 79 still appears with ASPM off, remove the parameter with sudo grubby --update-kernel=ALL --remove-args="pcie_aspm=off".

When Xid 79 still appears with runtime D3 and ASPM off, it's likely the GPU or the hardware on its PCI Express link. Contact the support organization for the GPU or the system with the Xid lines and any "PCIe Bus Error" lines from the failed boot, which is NVIDIA's investigatory action for Xid 79.

Related errors

Where ET Ducky shows it

I build ET Ducky. Its patch management updates Linux packages through apt, dnf, zypper, snap, and flatpak in rings. When Xid 79 starts after a driver or kernel update, you'd tag a few GPU hosts as your first ring and schedule updates to that tag before the rest. A patch is marked successful only when services come back up with no crash loops. A regression in a ring pauses the deployment. You can also open a Live Session shell from the browser without the need for a VPN and run the journalctl commands above on the host.

Sources

Driver and kernel updates in patch rings

Read how ET Ducky updates Linux packages through apt, dnf, zypper, snap, and flatpak in rings, and pauses the deployment when a ring regresses.

Read the patch management docs