← Home

Field Guide Collection · Reference

Triage
Decision Trees

For use during a problem, when you won't read prose. Start with the incident card, then follow the tree that matches the complaint. Each leaf points to the check that confirms it — the exact commands are on the Diagnostic Command Card — or to the likely cause. A leaf is a hypothesis, not a verdict: confirm it before you act. This card stands on its own; terms are in the Glossary.

1 incident card 3 trees leaf = hypothesis

Work top-down. The incident card is the same first fifteen minutes for any complaint; the three trees then diverge on symptom — can't reach, slow, intermittent. Follow one branch at a time and stop at the first leaf that matches what you observe, then switch to the Command Card to confirm it. If two branches seem to fit, a capture decides between them.

01

Incident Card — First 15 Minutes

Before reaching for any tool, run this sequence. Most of the value is in the first three steps — scope, what changed, and whether it ever worked — because they decide where to look.

  1. Scope it. One host? one subnet? one app? one site? everyone? This narrows where to look before you touch a tool.
  2. "What changed?" Recent config, patch, deploy, cabling, cert, or DHCP change? Most incidents trace to a recent change — check the change log first.
  3. Did it ever work? A new build points at a config or design problem; a regression points at something that changed.
  4. Capture a baseline. Before changing anything, note counters, take a pcap, and save logs — intermittent issues clear on their own and destroy your evidence.
  5. Bisect the path. loopback → own IP → gateway → next hop → remote → remote port. The first failure is the segment or layer to focus on.
  6. Rule out the cheap. Is DNS resolving right? Do large packets pass (MTU)? A link down? A full table?
  7. One hypothesis. State what you think is wrong and what you'd see if it were true — then test exactly that.
  8. Change one thing, reversibly. With a backout. Re-test. Then write the runbook entry.
Before any change to a live device

Confirm a rule permitting your own management access sits ahead of any new deny, use out-of-band access, and stage an automatic rollback (commit-confirmed / reload in) so a lockout self-heals instead of stranding the device.

02

Tree A — "Can't Reach X"

Resolve name before reachability, reachability before port. Ping proving L3 does not mean the service works, and ICMP being blocked does not mean the host is down — so don't stop at a failed ping.

Follow one branch; confirm each leaf with the Command Card

Can't reach HOST/service │ ├─ Does the NAME resolve correctly? (dig HOST ; compare to authoritative) │ ├─ No / wrong / NXDOMAIN ─────────▶ DNS problem │ │ • NXDOMAIN but should exist → negative caching, or querying a cache not authoritative │ │ • Wrong address → wrong resolver / /etc/hosts / split-horizon view │ │ • SERVFAIL → resolver/upstream/DNSSEC failure │ │ • Timeout → resolver unreachable (udp/53 & tcp/53 blocked? wrong server) │ └─ Yes, resolves correctly │ │ │ ├─ Does PING reach it? (ping HOST) │ │ ├─ No │ │ │ ├─ Can you ping your GATEWAY? │ │ │ │ ├─ No ─▶ local problem: link down, wrong IP/MASK/gateway, VLAN, │ │ │ │ │ APIPA (169.254 = no DHCP), gateway not on-link │ │ │ │ └─ Yes ─▶ routing/path problem beyond you: run traceroute, │ │ │ │ look for where it stops; check route table & next hop; │ │ │ │ consider firewall dropping ICMP (ping may be blocked though TCP works) │ │ │ └─ NOTE: ICMP may be filtered even when the service works — don't stop here. │ │ └─ Yes (ping works) ─ ping proves L3 only; keep going ▼ │ │ │ └─ Does the PORT respond? (nc -vz HOST PORT / Test-NetConnection) │ ├─ RST / "refused" ─▶ reachable host, NO LISTENER: │ │ service down, or bound to 127.0.0.1 not 0.0.0.0, or wrong port │ ├─ Timeout ─────────▶ FILTERED: firewall drop (check rules both directions; asymmetric │ │ routing into a stateful firewall; missing return/ephemeral rule) │ └─ Connects ─▶ L4 fine. Failure is higher up: │ ├─ TLS error? → cert/chain/SNI/clock (curl -v ; openssl s_client) │ ├─ Large transfers hang? → MTU / PMTUD black hole (ping -M do test) │ └─ App-level error? → application logs, dependency (DNS/NTP/auth) reachable?
03

Tree B — "It's Slow"

Split the question first: slow to start then fine is a resolution/handshake problem; slow throughout is latency or throughput — and those are different faults. High latency and low throughput have different causes and different fixes; adding bandwidth never fixes latency.

Separate latency from throughput before chasing either

"Slow" │ ├─ Slow to CONNECT/START, then fine? │ ├─ Yes ─▶ name-resolution or handshake latency: │ │ DNS lookup time (dig timing), TLS handshake, or a SYN being silently dropped+retried │ └─ No (slow throughout) ▼ │ ├─ Is LATENCY high, or is THROUGHPUT low? (different problems) │ ├─ High latency (RTT) │ │ ├─ Rises only UNDER LOAD ─▶ bufferbloat / queueing; link near saturation │ │ ├─ High at a specific hop to the end ─▶ real path latency (mtr); not a middle-hop ICMP artifact │ │ └─ Adding bandwidth won't fix latency. │ └─ Low throughput on a fast link │ ├─ Single stream slow, PARALLEL streams fast ─▶ TCP window / BDP limit (window scaling?) │ ├─ Throughput collapses with any loss on a high-RTT path ─▶ loss × RTT; find the loss (mtr) │ └─ Link actually saturated ─▶ capacity problem (check utilization, top talkers via flow data) │ └─ Errors present? (interface counters) └─ CRC/FCS + late collisions + slow ─▶ DUPLEX MISMATCH or bad cabling
04

Tree C — "It's Intermittent / Flaky"

The pattern is the diagnosis. Correlate with load, time, destination, packet size, or the set of affected hosts — each correlation points at a different mechanism. This is where the baseline capture from the incident card earns its keep, because the fault may not be present when you look.

Find the correlation; it names the mechanism

Intermittent │ ├─ Correlated with LOAD/TIME? │ ├─ Fails under load, recovers when quiet ─▶ resource exhaustion: │ │ conntrack/state table full, NAT/PAT source-port exhaustion, buffer drops, CPU │ └─ Periodic (regular interval) ─▶ DHCP renewal (T1/T2), route flap, STP reconvergence, │ cert/auth token expiry, scheduled job │ ├─ Affects SOME destinations/sizes, not others? │ ├─ Only LARGE packets/transfers ─▶ MTU / PMTUD black hole │ ├─ Only some destinations ─▶ asymmetric routing, one bad ECMP path, one bad anycast/LB node │ └─ Only after idle ─▶ middlebox state timeout (firewall/NAT idle-timeout); enable keepalives │ ├─ Affects SOME users/hosts, not others, same segment? │ └─▶ subnet MASK mismatch, duplicate IP, one bad switch port, MAC flapping, rogue DHCP │ └─ Whole SEGMENT drops/melts after a change? └─▶ L2 LOOP / STP failure (Ethernet has no TTL) — broadcast storm. Check recent cabling/switch adds.
05

How to Read These

A leaf is a hypothesis, not a verdict. Confirm it with the matching check on the Diagnostic Command Card before acting, and change one thing at a time so you can tell what worked. If two branches seem to fit, the capture decides between them. When a tree bottoms out with no match, widen the scope: re-ask "what changed?", and bisect the path again from a different vantage point — the fault may be on a return path or a device you haven't looked at.

Why the trees only go so far

These map symptoms to likely mechanisms; they don't prove one. The mechanism guides — how routing, DNS, TLS, NAT, and TCP actually behave — are what let you confirm a leaf rather than guess it. A branch is never a security judgment either: "reachable" is not "safe," and a filtered port tells you about policy, not intent.

Ported from the collection's networking source notes. Use with the Diagnostic Command Card (the exact commands per leaf) and the Network "Follow the Packet" guide (the mechanisms behind each leaf — the local-vs-remote decision, routing, DNS, TLS, NAT, TCP windows, MTU). A leaf names a likely cause, not a confirmed one; confirm with a capture before acting, and change one thing at a time with a backout. Terms are in the Glossary, and 00 · Start Here indexes the set.