The backup tool that never backed anything up

The alert arrived most mornings and said roughly the same thing every time: the server could not determine whether its kernel was up to date. On a machine hosting other people's businesses, that is not an email you ignore.

The obvious explanations were all wrong, and we worked through them in the order most people would.

What it was not

First guess was a broken software repository. The server pulls from more than a dozen of them, so one bad endpoint would explain an intermittent failure. Every repository listed fine when checked directly.

Second guess was DNS, because a sibling machine had recently been found with an empty resolver configuration and almost identical symptoms. This one resolved everything correctly.

Third guess was the package cache. Clearing it made the alert go away for a few hours, which felt like progress and was the most misleading result of the whole investigation.

What it actually was

The break came from watching the running process list at the moment the failure happened, rather than reading logs afterward. Two package manager processes were running at once. One was the control panel doing its scheduled update check. The other was a package install nobody had asked for.

They were deadlocking on the package database lock. When two processes fight over that lock, the loser reports a download failure that has nothing to do with downloads, which is why every mirror looked broken and none of them were.

The uninvited installs were coming from a backup plugin. It was reinstalling its own backup engine on a schedule, twice an hour and again every five minutes, each time reporting that the package was already installed.

The part that stung

Opening that plugin's admin panel produced the answer to a question we had not thought to ask. It showed a healthy count of active users, zero total backups and zero total jobs, above a banner explaining that initial setup had never been completed.

It had been installed months earlier and had never backed up a single file. Its entire contribution to that server was a package install loop that corrupted metadata and generated false security warnings.

It ran for three months, stored nothing, and its only measurable effect was breaking something else.

We removed it. The package manager went from hanging and failing to returning cleanly in under two seconds. The kernel warnings stopped and have not come back.

What we took from it

• The alert was accurate and the thing it named was fine. It said the kernel check failed, and the kernel check did fail, but the kernel was never the problem.
• Log files tell you what happened. The process list tells you what is happening. For anything intermittent, watch it live.
• A partial fix that works for a few hours is worse than no fix, because it sends you back to a theory that was already wrong.
• Count your backup systems. That machine turned out to be running four overlapping ones, and the noisiest was the one doing the least work.

The wider lesson is about installed-and-forgotten software. Nobody had touched that plugin since the day it went on. It was not malicious and it was not broken in any way its own interface would show you. It simply was not doing the job it was there to do, and it took a three day investigation into an unrelated warning to find that out.

If you run your own infrastructure, go and open the admin panel of every tool you installed and stopped thinking about. Check that it has actually done its job at least once.