Back to Blog

A Stop Job Is Running at Linux Shutdown and How to Find the Unit

Christopher 4 min read
referencelinux

"A stop job is running for" is a status line systemd prints on the console during shutdown or reboot while it waits for a unit to stop. systemd's source builds the line from the job type, the unit's description, the time elapsed, and the time limit, in this format.

A stop job is running for <unit description> (<elapsed> / <limit>)

The line appears once a job has run for five seconds. When more than one job is running, the line starts with a counter such as "(1 of 3)". The limit reads "no limit" when the job has no timeout.

What it means

The unit hasn't exited since systemd asked it to stop. For a service, TimeoutStopSec= sets how long systemd waits. It defaults to DefaultTimeoutStopSec=, which is 90 seconds in the system manager. systemd sends the service SIGTERM first. A service that hasn't stopped when the time runs out is terminated with SIGKILL.

How to find the cause

Show systemd's console messages during shutdown. On Fedora and RHEL the rhgb and quiet boot parameters hide them, so remove both from every boot entry and shut down again.

sudo grubby --update-kernel=ALL --remove-args="rhgb"
sudo grubby --update-kernel=ALL --remove-args="quiet"

Write down the text after "for" in the status line. That text is the unit's description. The unit's name can differ from it. On the next boot, find the unit name that goes with that description.

systemctl list-units --all | grep -F '<unit description>'

Read what that unit logged during the shutdown that hung. The -b -1 option selects the boot before the current one.

journalctl -b -1 -u <unit>

If only the current boot appears in journalctl --list-boots, the journal isn't kept across reboots. Set Storage=persistent in /etc/systemd/journald.conf so journal data is stored under /var/log/journal, then reproduce the hang.

Check the stop timeout the unit runs with. Time settings are exposed as properties ending in "USec", so TimeoutStopSec= appears as TimeoutStopUSec. Any TimeoutStopSec= set for that unit is in its unit file or a drop-in, which can be read with systemctl cat.

systemctl show -p TimeoutStopUSec <unit>
systemctl cat <unit>

Fixes

When the unit needs less time to stop than it's given, set a shorter stop timeout for that unit only. Run sudo systemctl edit <unit>, which creates a drop-in named override.conf, and add these lines.

[Service]
TimeoutStopSec=15s

A service still running when the 15 seconds run out is killed with SIGKILL, so pick a value long enough for its normal shutdown.

When many units are slow to stop, lower the default in a .conf file under /etc/systemd/system.conf.d/, but the new default applies to every unit on the host that doesn't set its own timeout.

[Manager]
DefaultTimeoutStopSec=30s

Related errors

Where ET Ducky shows it

I build ET Ducky. Reboots after patching follow a reboot policy (never, window_only, or immediate_if_required) and maintenance windows in each agent's local time zone. A patch is marked successful only when services come back up after the reboot with no crash loops. You can also open a Live Session shell from the browser without the need for a VPN and run the journalctl and systemctl commands above on the host.

Sources

Reboots inside your maintenance windows

Read how ET Ducky reboots after patching under a reboot policy and maintenance windows, and marks a patch successful only when services come back up.

Read the patch management docs