The SIEM that was blind for ten days

Starting point

My homelab runs Wazuh as its SIEM. Agents on the servers, containers and VMs send logs and security events to a central manager. Every morning a script pushes a summary of the last 24 hours to my phone, and an uptime monitor continuously checks that the dashboard is reachable.

What happened

On 22 June 2026 I deployed a set of custom detection rules and restarted the Wazuh manager to load them. The restart looked clean: the service reported “running” and the dashboard was up.

In fact, the component that accepts agent connections had hung during start-up. From that moment on, not a single event from the infrastructure arrived. 16 of 17 agents were shown as disconnected.

Nobody noticed for ten days.

Why nobody noticed

Three controls were in place. None of them fired:

• The uptime monitor checked whether the dashboard answered over HTTPS. It did.
• The daily report counted the events of the last 24 hours. Zero events looked like a quiet day.
• The service status said “running”. The process was alive; it just was not working.

All three measured whether the system was there. None measured whether it was working.

How it surfaced

On 2 July I ran a full audit of the homelab using Claude Code: read-only data collection, analysis from several angles, and every finding re-measured live before it was recorded. The only critical finding: no agent telemetry for ten days.

Restarting the manager cleared the fault, and all twelve always-on agents came back as active. The cause was in the log for 22 June: start-up had stalled right after reading the key file, before the first connection was accepted.

What I changed

1. A watcher that measures whether data arrives. The daily report now checks every always-on agent individually and sends a separate high-priority alert as soon as one is not active. Laptops and desktops are deliberately excluded, as they are not always switched on.
2. Check the agents after every manager restart. A rule deployment is a change to a production system and gets verified like any other.

The watcher had a blind spot of its own

In September, while refining it, I found that the new watcher had a weakness of its own. The script aborted on any error (set -e). It first queries the search database for the report and only then checks the agents, through a different interface. If the database has a problem, as had already happened once in the homelab, the script aborts before the agent check runs. No alert, no error mail. The watcher would have gone silent in exactly the kind of situation it was needed for.

Since 20 September the check runs independently of the report and from two directions:

• An agent is not active although its machine is running → alert.
• A machine is running but has no agent record at all → alert.

Without the second direction, a deleted agent would have stayed invisible. The only list still maintained by hand is the one of excluded end-user devices. If something is forgotten there, the result is a false alarm, loud and correctable, rather than a silent blind spot.

What I took away

• “Zero events” is not good news; it is a reading that needs an explanation.
• A process that runs is not the same as a service that works. Monitor the outcome, not the status.
• A watcher that hangs from the same branch as what it watches falls with it.
• False alarms can be corrected; silent gaps cannot. When in doubt, build it loud.

What this means for a business

A SIEM whose data intake nobody monitors creates a false sense of security. The dashboard is green, the report shows a quiet day, and the board assumes someone is watching. In reality the organisation is blind, and that is exactly when an attacker can work unnoticed.

The question to ask your own IT team or provider is simple: How would we notice that our SIEM has stopped seeing anything, and how quickly? If the answer is “on the dashboard” or “in the monthly report”, a control is missing. It costs little: a daily check that every critical source is still delivering, with an alert that does not depend on the system being monitored.

There is a second point: changes to security systems, such as new rules, updates or restarts, are changes like any other. They need a check afterwards, not just approval beforehand.