Analyst Field Guide · Revision 1
How to drive an incident or recurring failure back to the condition that allowed it — and prove you got there. Built for security analysts; the methods apply to any high-impact problem.
A root cause is the origin that, if corrected, stops the problem recurring. Not the symptom you saw. Not the first plausible answer. In security, that almost always means the control failure or access path that let an event happen — never the malware or the alert itself. You are not finished when the problem is fixed; you are finished when you can name what allowed it and have closed that.
Every RCA failure traces to confusing these four layers. Learn to name which one you are looking at.
Eradicating the symptom (the malware, the alert) while leaving the root cause (the access vector) open — so the intrusion returns next week with a new payload.
RCA has a real time cost. Spend it where recurrence or impact justifies it; run a normal fix everywhere else.
Seven phases, with a loop: new evidence sends you back, never forward on a guess.
The methods only work on good inputs. A vague statement produces a vague cause. Write one or two factual sentences — no cause guessed yet — covering what, where, when, scope, and impact.
| Weak | Field-ready |
|---|---|
| "The server keeps going down." | "WEB-03 returned HTTP 503 for 40 min, 2 Jun 14:10–14:50 UTC; ~2,000 sessions affected." |
| "We got hacked." | "An external IP authenticated to the VPN with a valid employee credential at 14:10 UTC and queried the customer DB; scope under investigation." |
[What] occurred on [system] during [timeframe], affecting [scope]. Impact: [measurable effect]. Detected by [source].
Pin the problem down four ways, and state what it is not. The boundary between IS and IS-NOT exposes the distinguishing factor — usually a change.
| Dimension | IS | IS-NOT | Points to |
|---|---|---|---|
| What | 503s on WEB-03 | WEB-01 / WEB-02 | something specific to WEB-03 |
| Where | EU region | US region | a region-scoped change |
| When | since 14:10, 2 Jun | before the 14:00 deploy | the deploy as a suspect |
| Extent | ~2,000 sessions | not 100% of traffic | a partial / conditional cause |
Illustrative values.
Before reasoning about why, establish what happened, in order, from evidence: logs, metrics, change records, access records, alerts. Tag every entry by confidence.
confirmed observed here, evidence attached suspected consistent, unproven — state what would confirm theoretical possible, not tied to this case
Logging in, rebooting, or re-imaging alters state. Capture volatile and forensic evidence before remediation, and record exactly what you ran and when. A help-desk cleanup can destroy the evidence the analysis depends on.
For incidents the timeline is much of the analysis. At each event ask two things: what caused it, and what control should have caught it and didn't. Each gap is a candidate root cause.
Every method below is one of three reasoning modes in disguise. Knowing which mode you're in tells you what a step can and cannot prove — and most RCA failures are a guess (abduction) that never got tested (deduction).
From a surprising observation, propose the cause that would best explain it. This is hypothesis generation — the first "why", every bone on a fishbone. Fast, but the "best" explanation is often merely the most familiar; hand its output to deduction.
"If the root cause is X, then we must observe Y." You derive a prediction the evidence has to satisfy, then check it. This is hypothesis testing — the forward-logic and removal tests. It can falsify a wrong candidate outright.
Generalize across many incidents: "these 40 all involved an unpatched internet-facing service." This is what Pareto and trend analysis do. A pattern is not proof of cause — correlation isn't causation, and one new case can break the rule. Confirm with deduction.
Reasoning from a similar past incident is fast but carries the solution-context fallacy: the prior context may differ. Treat the analogy as a source of hypotheses, not an answer.
The workhorse for drilling a single thread. Take a confirmed fact and ask "why" repeatedly, each answer becoming the next question, until you reach a systemic, controllable cause. "Five" is a guide, not a quota.
Interacting or parallel causes (it is single-thread by nature) and chains that are plausible but unevidenced. Switch to a fishbone or fault tree when the cause isn't a single line.
Use when you don't yet know which direction to drill. It surfaces candidates across categories so you don't tunnel on the first idea.
A fishbone produces hypotheses, not conclusions. After building it you must test each surviving candidate against the timeline and discard what the facts don't support. "It's on the diagram" is not "it's a cause." Then drill the survivors with 5 Whys or a fault tree.
Use when the failure required several conditions to coincide — common in security. Top-down and logic-based; it makes AND vs OR explicit, which is what reveals your highest-leverage fix.
Use when you have many incidents and must decide which cause to attack first. It tells you which root to pursue, not what it is — pair it with a 5 Whys on the top bars.
Two focused techniques worth keeping in the kit.
When something worked yesterday and failed today, the cause is usually a change. Compare the failure state against a known-good baseline and list every difference — deployments, config edits, patches, new accounts, network changes. Each difference is a suspect. This is the fastest route for "it was fine until…" problems and pairs directly with change/version records.
Model the defenses (barriers) that should have stopped the harm and identify which were missing, bypassed, or failed. In security terms: each control between the threat and the asset is a barrier; the analysis maps which ones the event passed through untouched. The failed/absent barriers are candidate roots — and the list of barriers that held is your honest statement of residual defense.
Two models for reasoning about layered defenses — they explain why incidents need several failures at once, and where each control sits.
Each defensive layer has holes — latent weaknesses and active failures. Defense-in-depth works because the holes rarely line up. An incident is the moment they do, letting a hazard pass clean through every layer.
A fault tree and an event tree joined at a central top event. Left: threats pass through preventive controls toward the event. Right: from the event, mitigative controls limit the consequences. It shows, for one hazard, both how it happens and how its impact is contained — and exactly which control sits where.
| Situation | Reach for | Why |
|---|---|---|
| Cause is a clear single thread | 5 Whys | Fast linear drill |
| Direction unclear, need breadth | Fishbone | Generates candidates without tunneling |
| Several conditions had to coincide | Fault Tree | Makes AND/OR and cut sets explicit |
| Many recurring incidents | Pareto | Prioritizes the vital few |
| "Worked yesterday, broke today" | Change Analysis | Isolates the introduced difference |
| Asking which defense failed | Barrier Analysis | Maps missing/bypassed controls |
| Any incident | Timeline first | Order and control-gaps carry the meaning |
They chain, they aren't alternatives: statement + timeline → fishbone to enumerate → narrow to evidenced candidates → 5 Whys / fault tree to drill → validate → act.
RCA is reactive: it analyzes a failure that already happened. FMEA (Failure Mode & Effects Analysis) is its forward-looking counterpart — enumerate how a system could fail, score each mode by severity × occurrence × detectability (the RPN), and fix the high scorers before they bite. Use FMEA in design and hardening; use RCA after an event.
Run every candidate through these five tests. Fail one and you stopped too early.
| Test | Question | Fail means |
|---|---|---|
| Removal | If corrected, would the problem recur? | Still recurs → not the root |
| Control | Can you actually influence it? | "Human nature", "it rained" → go up to the controllable factor |
| Forward-logic | Does the chain read "X, therefore Y" without leaps? | A leap → chain is wrong |
| Substitution | Would a different competent person in the same system hit it? | Yes → systemic, not the individual |
| Multiple-root | Confirmed there isn't a parallel cause? | Unchecked → likely incomplete |
It survives all five tests, is evidenced, and is within your power to fix. The substitution test is also what keeps the analysis blameless.
Simple, linear failures usually have a findable root. Complex system failures often do not — they emerge from many interacting contributing factors, and naming a single "root cause" can be a hindsight simplification that ends the analysis too early. This is contested: the systems-safety view (Reason, Dekker, Cook) holds that complex failures rarely have one cause. Practical rule: for a linear fault, find and fix the root; for a complex incident, enumerate the contributing factors and fix the set — report them honestly rather than forcing a single root.
Each confirmed root cause earns two actions:
Every action carries an owner, a date, the resources needed, and the risk of the action itself. For preventive/hardening actions, state the threat model they assume — a defense is only evaluable against a defined adversary.
A root cause with no owned, dated, verified action is documentation, not analysis. RCA is done when recurrence stops — confirmed in monitoring — not when the report ships.
| Section | Contents |
|---|---|
| Problem statement | One precise paragraph: what, scope, impact, timeframe. |
| Timeline | Key events in order with evidence refs; for incidents, the attack/event chain. |
| Root cause(s) | Stated plainly, distinguished from contributing factors. For a breach, name the control failure. |
| Contributing factors | Conditions that made it possible or worse. |
| Corrective actions | Immediate fixes / eradication — owner, date. |
| Preventive actions | Systemic changes — owner, date, assumed threat model. |
| Residual / out-of-scope risk | What remains after the actions. |
| Verification | How each action will be confirmed effective. |
So RCA doesn't become write-only:
| Metric | What it tells you |
|---|---|
| Recurrence rate | Same root cause reappearing — the primary signal RCA is working. |
| % incidents with verified actions | Whether findings become closed, confirmed fixes — not just identified ones. |
| Action closure time | Whether fixes ship before the next occurrence. |
They measure process health, not security. A low recurrence rate is not evidence the environment is secure.
Blameless does not mean no accountability; it means the response fits the behaviour. Distinguish three:
| Behaviour | Example | Response |
|---|---|---|
| Human error | slipped, misread, fat-fingered | console; fix the system that made the error easy and consequential |
| At-risk | took an unsafe shortcut, risk not perceived | coach; remove the incentive that made the shortcut look reasonable |
| Reckless | conscious disregard of a substantial known risk | accountability / discipline |
Only conscious recklessness warrants discipline; treating ordinary error punitively destroys the honest reporting RCA depends on. These are management responses — keep them separate from the technical cause analysis.
Statement → preserve evidence + timeline → enumerate (fishbone) → narrow to evidenced → drill (5 Whys / fault tree) → validate (5 tests) → CAPA → verify it holds.
Removal · Control · Forward-logic · Substitution · Multiple-root.
Abduce (generate) → Deduce (test/falsify) → Induce (find patterns). A guess you never tested isn't a root cause.
Swiss Cheese — holes align across layers; find a factor at each. Bowtie — preventive controls left of the event, mitigative right.
Methods shown — 5 Whys, Ishikawa/fishbone, Fault Tree Analysis, Pareto, timeline/event sequencing, Change Analysis, Barrier Analysis — are established, widely documented techniques; the process here is synthesized from them and from the source notes (Root Cause Analysis — ProjectManager / Peter Landau). All worked examples and chart values are illustrative, not real incidents or measured data. Framework- and compliance-specific identifiers (e.g. ATT&CK technique IDs, control numbers) are intentionally omitted — verify and cite your organization's currently adopted revision before mapping findings to one, as identifiers change between revisions.