Whyfire
Self-hosted alert investigation

Every alert is a small fire. Whyfire finds the point of origin: what changed first, where, and how it spread. On-call gets the cause, the evidence and the next step before anyone opens a dashboard.

Become a design partner See how it finds the origin

Runs on your own server. Built on Prometheus, Alertmanager and Loki.

FIRE REPORT · INC-01423 MIN AFTER FIRST ALARM

Checkout down: database primary saturated

Origin
db-primary · 13:58
4 minutes before the first alarm
Cause
A report query holding row locks on orders
Spread
checkout-api-1, checkout-api-2, db-replica-1 · 7 alarms, 1 incident
Next step
End session 4411 after checking with its owner; add a statement timeout so it can't recur.
1
Report query holding row locks
0.82
2
Connection pool too small
0.11
3
Storage latency
0.04
Was the origin right?YesNo
SAMPLE DATA
Point of origin

Fire investigators work backwards.
So does Whyfire.

The alert that fires first is rarely where the fire started. Whyfire looks at everything that moved around the time of the alarm and asks which change came first.

  1. 01

    Compare with normal, for this hour

    Every signal is measured against its own usual level at this time of day over the past week, so the nightly backup doesn't look like a fire.

  2. 02

    Find what moved first

    Across every host, metric and log stream, the strongest changes are lined up by when they started. The earliest one upstream of the alarm is the prime suspect.

  3. 03

    Rule out the usual suspects

    Changes that happen at the same time every day are marked routine, and past incidents and silences on the same alert are pulled in for context.

  4. 04

    Only then, the write-up

    The AI writes the report from that evidence. The ranking stands on its own, so you can check the working, not just the prose.

  1. Report query startsreporting_svc · nothing alarming yet
  2. Blocked sessions start climbing Origindb-primary · 4 → 96 in four minutes
  3. Alarm: DbBlockedSessionsHighdb-primary
  4. Connection pool fullcheckout-api, one minute after the locks: a symptom
  5. Alarm: HttpErrorRateHigh ×2checkout-api-1, checkout-api-2
  6. Fire report posted to Teamsone incident, origin ranked first
  7. Alarm: ReplicationLagHighdb-replica-1 · joins the same incident
Why it's different

One fire, not twenty smoke alarms.

Plenty of tools now bolt an AI summary onto your alerts. Whyfire shows its working, keeps score, and lives on your own server.

One incident per fire

Alarms that start together and share a cause become one incident with one report, not a card per host.

Evidence before prose

Causes are ranked by statistics first and written up second. If the write-up and the ranking disagree, you can see it.

It keeps score

On-call answers "was the origin right?" in one click, and a weekly benchmark tests the ranking against incidents with a known cause.

Alarms that re-arm

Silence an alert against a ticket and Whyfire keeps it quiet while the ticket is open, then re-arms it when the ticket closes, in case the fix didn't hold.

Remembers past fires

Similar past incidents, earlier silences and their tickets go into every report: "this happened on 6 Oct, same query".

Your building, your alarm

One appliance on your own server. Metrics and logs stay with you, on Prometheus, Alertmanager and Loki that your team already knows.

Install

Up in an afternoon.

One master node, like a VPN appliance. One line per server, like the big SaaS agents.

01 · master

Install the appliance

On one Ubuntu server or VM. It brings its own metrics, logs, alert routing and investigator.

$ sudo bash install.sh
02 · agents

Add your servers

The appliance prints a one-line command with an enrolment token. Each host appears within a minute.

$ curl -fsSL http://master:8088/agent.sh \
    | sudo bash -s -- --token …
03 · alarms

Get fire reports

Point it at Teams or Slack. The next time something breaks, the report arrives with the alarm.

# 7 alarms → 1 incident
# origin: db-primary 13:58
Built for

Small teams with real fires.

In-house infrastructure teams

Two to twenty people looking after a mix of modern and legacy servers, already drowning in alert cards.

Managed service providers

Many small estates, thin on-call rotas, and every minute of diagnosis billed or lost.

Sites where data can't leave

Public sector, finance and their suppliers, where SaaS observability is off the table.

Early access

Seen enough smoke?

Whyfire is in development. Join the waitlist to hear when it's ready, or tick the box to become a design partner: you get the appliance free while it's built, and we get your feedback and your votes on the ranking.

Rather talk first? hello@whyfire.io

We only use this to contact you about Whyfire. Privacy notice