The security updates that never ran

Starting point

Every Linux system in my homelab is meant to install security updates on its own. On Debian and Ubuntu that is the job of unattended-upgrades: once a day it fetches the package lists, picks the updates that come from the security archive, installs them and writes a line to its log. Feature updates stay manual. That was the rule, and the documentation said it was in place on ten systems.

What happened

In September 2026 an authenticated vulnerability scan flagged an outdated video codec library on the document-management container. Debian had already published the fix. Installing it by hand took a minute; the more interesting question was why the automatic updates had not done it.

The log answered with a clean entry from that very morning: “No packages found that can be upgraded unattended.” At the same time, six security updates were waiting on the same machine. The next morning, after another clean run, they were still waiting.

Going back through the logs made it worse. They start in early April 2026, when the automatic updates were set up; nothing older had been rotated away. In the almost six months since, on the hypervisor and the affected containers, the job had not installed a single update. Not one day. It had never worked.

Why nobody noticed

The job never failed. It started on schedule, read its configuration, compared the available updates with the sources it was allowed to use, found no match, and reported that there was nothing to do. Every morning, on every affected system.

The cause was a single line. The configuration named the permitted source by its release codename, but compared that name with the archive name (apt calls it the suite). Debian names the security archive by role (stable-security, oldstable-security) and keeps the codename in a separate field. The rule looked for an archive name that does not exist. The packages were there; the filter let none of them through.

# the rule, as configured
Unattended-Upgrade::Allowed-Origins {
    "${distro_id}:${distro_codename}-security";
};
# ...as unattended-upgrades reads it
o=Debian,a=bookworm-security

# the security archive, as apt sees it
o=Debian,a=oldstable-security,n=bookworm-security

A dry run with debug output shows the filter at work. The tool finds the update, checks where it comes from, falls back to the installed version and reports that there is nothing to do:

Checking: openssl ([<Origin ... archive:'oldstable-security' origin:'Debian' ...>])
adjusting candidate version: openssl=3.0.20-1~deb12u2
No packages found that can be upgraded unattended and no pending auto-removals

My first explanation was that a new Debian release had renamed the archive and broken a rule that used to work. It sounded plausible and would have made a tidier story. Debian’s own naming did not support it: the security archive’s name states the role, not the release, so the rule could not have matched under that scheme. The logs are consistent with that: not a single installation in six months.

A green log, no errors, no alerts. Anyone checking would have found a job that runs.

How far it went

Once the pattern was clear, I checked every system individually instead of trusting the list in the documentation.

5Rule matched nothing

2Installed, never on

4Never set up

3Working

No automatic security updates (11) Working as intended (3)

• Five systems, including the hypervisor, had the broken rule described above.
• Four had never had automatic updates set up at all. They were simply missing from the list.
• Two had the package installed but never switched on, with a configuration file that had been written with doubled quotation marks and was read as nonsense.
• Three worked: the Ubuntu machines, whose archive is named after the codename, and one system that used the distribution’s default configuration.

In total, 80 security updates were outstanding. The largest backlog was on a web application container (twenty, mostly the PHP runtime) and on the development machine (thirty-seven). All of them are installed now; every system shows zero outstanding security updates.

The count had a blind spot of its own

My first measurement said the systems without working automatic updates had nothing outstanding. That was wrong. Without the daily job, nobody refreshes the package lists, and the check for pending updates compares against a list that is weeks old. After a manual refresh, four of them showed twenty, three, three and thirty-seven.

A second, smaller trap: two machines run with a German locale. The log line I was searching for in English was there, in German. Every check now forces a neutral locale.

What I changed

1. A rule that matches the right field. Every Debian system now matches the security source on its codename field instead of its archive name, which is also how the distribution’s own default does it, and which does not change when a release moves from stable to oldstable. The fix is an additional file; the original configuration stays untouched and can be restored by deleting one file.
2. One script for all systems. It installs and enables automatic updates where missing, restricts them to security updates, adds the rule and shows what it would install. It runs as a dry run by default and changes nothing without an explicit flag.
3. Installed is not applied. A patched library only takes effect once the services using it restart. After patching a shared library I list the processes still holding the deleted old version and restart those services; two small containers were simply restarted whole.
4. The measure of health is what is missing. Whether automatic updates work is no longer read from their log, but from the number of security updates still outstanding a day after one is published, counted against freshly fetched lists.

The rule that replaced the broken one, and the two checks, for anyone who wants to run them:

# match the security source on its codename field, not its archive name
Unattended-Upgrade::Origins-Pattern {
    "origin=Debian,codename=${distro_codename}-security,label=Debian-Security";
};

# health: security updates still outstanding, counted against fresh lists
apt-get update -qq && apt list --upgradable 2>/dev/null | grep -c -- -security

# installed is not applied: processes still running a deleted library
grep -l '\.so.*(deleted)' /proc/[0-9]*/maps | cut -d/ -f3 | xargs -r ps -o comm= -p | sort -u

What I took away

• A clean log proves that a job ran, not that it did anything.
• “Nothing to do” deserves the same suspicion as “zero events”. It is a reading, not a result.
• An inventory that is maintained by hand drifts. The documentation said ten systems; the truth was fourteen, and eleven of them were not covered.
• Check a measurement against its own inputs. A count of pending updates is only as current as the list it counts against.
• A plausible cause is not a proven one. The tidy explanation, a release that broke a working rule, did not survive a look at how Debian actually names its archives.

What this means for a business

“Automatic security updates are enabled” is one of the most common answers to the patching question, and one of the least verified. It describes a setting, not an outcome. A configuration that quietly matches nothing, a machine that was never added, a job that was never switched on: none of these produce an error, and all of them leave systems unpatched while every report stays green.

The question to ask your own IT team or provider: How many security updates are outstanding on each system right now, and how do we know the number is current? If the answer is a policy (“updates are automatic”) rather than a figure per system, the control has not been measured.

It costs little to close. A daily count of outstanding security updates per system, with an alert when a number stays above zero for more than a day or two, turns a setting you believe in into a control you can see working. And after patching a shared library, the job is not done until the services using it have restarted.