Finding Why a Linux Service Stopped After an Update

A service on an Ubuntu or Debian server shows as inactive the morning after a round of updates, and its own log has no error in it. Restarting it fixes the symptom, but the next update can stop it again. This post covers how to tell a crash from a stop request, how to find what asked systemd to stop the service, and how to check every service after an update so the next one is caught in minutes.

Find out whether it crashed or was stopped

I'd start with the unit's status. Replace haproxy.service with your own unit in every command below.

systemctl status haproxy.service --no-pager

A unit that shows inactive (dead) was stopped cleanly. A unit that crashed or failed to start shows failed with a result such as exit-code or signal. When the log lines under the status show "Stopping haproxy.service" and a process that exited with code 143, that process received SIGTERM (128 plus signal 15). systemd sends SIGTERM when something asks it to stop the unit.

Read the unit's own journal for the day to get the exact second of the stop.

journalctl -u haproxy.service --since today -o short-iso --no-pager | tail -n 20

A restart that fails leaves a different trail. The journal shows "Stopping", then "Starting", then an error from the service's own start command, such as a configuration test that failed. That points at the service's configuration or at something it depends on being unavailable at that second.

Check what apt ran at that minute

Every apt and unattended-upgrades run writes a block to /var/log/apt/history.log with a Start-Date, the command line, the packages it changed, and an End-Date. Older months are rotated into compressed files, so search them with zgrep.

zgrep -h -A4 "Start-Date: 2026-10-10" /var/log/apt/history.log*

When the Commandline line shows /usr/bin/unattended-upgrade, the automatic updates ran it. Compare its Start-Date and End-Date with the time of the stop. Then check whether the service's own package was in that run.

zgrep -h " upgrade haproxy" /var/log/dpkg.log*

A package upgrade usually restarts its own service from the package scripts. The unattended-upgrades runs also write the full dpkg output, including any service restarts, to /var/log/unattended-upgrades/unattended-upgrades-dpkg.log. Search that file for the unit name.

sudo grep -n "haproxy" /var/log/unattended-upgrades/unattended-upgrades-dpkg.log

Run systemctl list-timers apt-daily-upgrade.timer to see when the automatic upgrade last ran and when it runs next. The timer runs at a set time with a random delay, so the stop time can move from day to day.

Check needrestart on Ubuntu 24.04 and later

A library upgrade can restart a service whose own package didn't change. On Ubuntu 24.04 needrestart restarts the affected services directly after apt finishes, in unattended-upgrades runs too. Ubuntu documented the change in needrestart changes in Ubuntu 24.04. A glibc update, for example, restarts most long-running services in one systemctl restart call. That call is in unattended-upgrades-dpkg.log with the full list of units it restarted.

A service that starts before something it needs at that second, such as name resolution, fails its start and stays down. When a service has to stay out of those restarts, add it to $nrconf{override_rc} in a file under /etc/needrestart/conf.d/ and restart it yourself in a maintenance window.

$nrconf{override_rc}{qr(^haproxy\.service$)} = 0;

Read everything logged in the minute before the stop

When apt didn't run at that time, read every unit's journal lines from the minute before the stop. The -o cat output leaves out the hostname, which is useful if you post it on a forum.

sudo journalctl --since "2026-10-10 10:04:30" --until "2026-10-10 10:05:38" -o cat

A stop that someone ran by hand with sudo shows up as a sudo line with the command it ran. Search for it directly.

sudo journalctl -t sudo --since "2026-10-10 10:00" --until "2026-10-10 10:10" -o cat | grep systemctl

Check the timers that fired in that minute with systemctl list-timers --all, and check crontab and /etc/cron.d for a script that stops services before a backup or a certificate renewal. A configuration management run from Ansible or Puppet also shows up as its own lines in the journal for that minute.

Prevent the next one

Adding Restart=on-failure to the unit doesn't help with this. systemd doesn't restart a service when its death is the result of a stop or restart that systemd performed. The fix depends on what asked for the stop.

When the stop came from unattended-upgrades, exclude that package with Unattended-Upgrade::Package-Blacklist in /etc/apt/apt.conf.d/50unattended-upgrades and upgrade it by hand. When it came from needrestart, use override_rc as above. When it came from a script or a person, fix the script or the runbook.

Also check every service after each update. Save the list of running services before the update and compare it after.

systemctl list-units --type=service --state=running --no-legend | awk '{print $1}' | sort > /tmp/running-before.txt
sudo apt upgrade
systemctl list-units --type=service --state=running --no-legend | awk '{print $1}' | sort | comm -23 /tmp/running-before.txt -

Any unit the last command prints was running before the update and isn't running now. Run systemctl --failed as well to list units that failed to start.

Where ET Ducky shows it

I build ET Ducky, and its Linux agent reads process exec and exit events from eBPF tracepoints. The systemctl, needrestart, or package script process that ran in the minute before the stop is in that event record, along with the service's own processes exiting.

For patching, ET Ducky runs apt updates in rollout rings. You'd tag a few servers as your first ring and schedule updates to that tag before the rest. A patch is marked successful only when the kernel event stream shows services back up with no crash loops. If the first ring regresses, the deployment pauses before the next ring. You can also open a Live Session shell from the browser without the need for a VPN and run the journalctl and zgrep commands above on the host.