Technician's Field Guide · Networking
Learn to reason about any network problem by tracing one packet from source to destination and back. The mental models, the mechanisms behind them, and a repeatable method — with worked examples, so the ideas stick.
Nothing on a network is magic — everything is deterministic and traceable. "Flaky," "weird," and "random" describe your current ignorance, not the network. Every packet went somewhere for a reason you can find. The master skill is to follow the packet: mentally walk one from source to destination and back, and the point where your walk goes uncertain is exactly where to look. This guide builds that skill layer by layer.
Six mental models do more for troubleshooting than any single protocol fact. Internalize these first; the rest of the guide is mechanism to hang on them.
It's rarely DNS — but rule it out early because it's cheap. Same for MTU. High-frequency, low-cost checks (DNS, MTU, a dropped link, a full table) come before deep investigation.
The single most misunderstood thing in networking is what the subnet mask actually does. It isn't decoration next to the IP — it's half of a decision the host makes on every packet: is the destination local (deliver directly) or remote (send to the gateway)?
Find the "interesting" octet (the one where the mask isn't 255 or 0). The block size there is 256 − mask_value, and subnet boundaries step by that size. That's all you need.
Two hosts on the same wire with different masks reach each other asymmetrically or not at all: one treats an address as local and ARPs for it directly; the other treats it as remote and sends to the gateway. "A can reach B but B can't reach A" on one segment is a mask mismatch far more often than a firewall or routing bug. Private ranges (10/8, 172.16/12, 192.168/16) are an addressing scope, not a security boundary — and a 169.254.x.x address means "no DHCP was reached," not an IP conflict.
IPv6 isn't "IPv4 with more digits"; several differences change how you troubleshoot. Addresses are 128-bit, written as hex groups with :: compressing one run of zeros (2001:db8::1), and the subnet is almost always a /64. Every interface also carries a link-local address (fe80::/10) for on-link talk. There is no ARP — IPv6 uses NDP (Neighbor Discovery over ICMPv6), so unlike IPv4, blocking ICMPv6 breaks basic connectivity, not just ping. Addresses come from SLAAC (the host builds its own from Router Advertisements), DHCPv6, or both — so a host commonly holds several at once (a link-local plus one or more global). And normally there's no NAT: every host is globally routable, which makes the firewall — not address scarcity — the security boundary (the same point the box above makes about private ranges). The gotcha that bites help desks: most networks are dual-stack (v4 and v6 at once) and clients prefer whichever connects first ("Happy Eyeballs"), so a service you're debugging over IPv4 may actually be reached over IPv6 or the reverse — and firewall rules written only for v4 often leave v6 wide open. Confirm which family is really in use (ping -6; note browsers need brackets, https://[2001:db8::1]) before concluding anything.
Before the mask can decide anything, the host needs an address — and usually gets it from DHCP, which hands over the IP, mask, default gateway, and DNS resolver in one exchange. A large share of "I have no network" reports are DHCP failures in disguise: an APIPA address, a wrong gateway, or a resolver the client should never have received.
Because the client has no address yet, it starts with an L2 broadcast — and routers don't forward broadcasts, so a client on a different subnet from the server needs a relay (an ip helper-address) to carry the request across. That's the classic split symptom: hosts on the server's own subnet get leases, but a remote VLAN gets APIPA → a missing or misconfigured relay, not a dead server.
An address is a lease, not a gift: the client renews at ~50% of the lease term (unicast to its server) and, if that fails, rebinds at ~87.5% (broadcast to any server) — so intermittent loss on a regular cadence can be a renewal-timing problem, not full expiry. (Those fractions are RFC defaults; a server can override them — verify.)
169.254.x.x (APIPA) means the client got no lease at all — it never reached a server. Read it as a path problem toward the server (link, VLAN, relay, or an exhausted pool), not an address conflict. And a valid address where nothing works is usually a missing or wrong option — the gateway (option 3) or DNS (option 6) — because an address alone isn't enough. Watch the exchange with tcpdump -ni IF port 67 or port 68: no Offer = no server reached; a second, unexpected Offer = a rogue DHCP server (see §4).
Below routing sits the layer that actually moves frames on the wire. A switch learns by recording source MAC → port as frames arrive and ages entries out; for a destination MAC it hasn't learned, it floods the frame out every port in the VLAN. Get this layer wrong and the symptoms are dramatic and fast.
A single switch is one broadcast domain until you add VLANs — each VLAN is its own broadcast domain, and traffic between VLANs must be routed. "Two devices on the same switch can't talk" is a VLAN-membership or trunk question before it's anything routed.
Ethernet frames have no TTL — nothing kills a looping frame, so a single broadcast multiplies forever and saturates the segment (a broadcast storm). Spanning Tree (STP/RSTP) exists solely to block loops. Signature: a whole segment "melts" — 100% switch CPU, total loss — right after someone patched a cable or added a switch. Suspect a loop before anything else.
L2 sits below most monitoring, so its threats are easy to miss. The standard mitigations (feature names are common across vendors; exact syntax varies — verify on your hardware):
| Threat | Mechanism | Mitigation |
|---|---|---|
| ARP spoofing / MITM | Forged ARP rebinds traffic to the attacker | Dynamic ARP Inspection |
| Rogue DHCP | Attacker hands out gateway/DNS | DHCP snooping (trust only known ports) |
| MAC flooding | Fill the MAC table → switch floods everything | Port security (limit MACs/port) |
| STP manipulation | Forged BPDUs reshape or blackhole L2 | BPDU guard / root guard on edge ports |
| VLAN hopping | Abuse of auto-trunking / double-tagging | Disable auto-trunk; set the native VLAN deliberately |
DHCP snooping's binding table is also what Dynamic ARP Inspection and IP Source Guard build on, so it earns its keep beyond DHCP. As always, none of this makes L2 "secure" — each control addresses a specific forgery and leaves the rest of the surface (e.g. IPv6 rogue Router Advertisements, which need RA Guard) untouched.
Routing is per-hop and destination-based. No single device knows the whole path; each one answers only "given this destination, what's my next hop and egress interface?" and forwards. The selection rule is Longest Prefix Match (LPM): among all matching routes, the most specific (longest) prefix wins — evaluated before metric or administrative distance.
The default route (0.0.0.0/0) has the shortest possible prefix, so it only wins when nothing more specific matches — the catch-all, not a preference.
Two tie-breakers sit below LPM, only for equally-specific routes: administrative distance chooses between different sources for the same prefix (a Cisco concept — the defaults are vendor-specific and configurable, so verify them), and metric chooses within one protocol. The control plane's full list of candidates is the RIB; the optimized table the hardware forwards with at line rate is the FIB.
Everything above assumes packets flow freely. When a link saturates, devices queue — and Quality of Service decides what goes first and what gets dropped. Traffic is marked (the DSCP bits in the IP header), classified into queues, and scheduled so latency-sensitive flows (voice, video, real-time control) are served ahead of bulk transfers (backups, file copies) that don't care about a few milliseconds. Two things worth knowing at the help desk: QoS only acts at the point of congestion, so "add QoS" won't fix a problem that's really a saturated uplink or a duplex mismatch; and DSCP markings are only honored inside a domain that trusts them — most ISPs and many inter-org links rewrite or ignore them, so end-to-end QoS across the public internet is not something you can assume. It prioritizes contention; it does not create bandwidth.
DNS turns names into addresses, and it's one of the highest-frequency "the network is broken" root causes — though most of those are caching or resolver-selection problems, not DNS being down. The key mental model: every layer caches independently, and the authoritative server is the only source of truth; everything else is a cache of it.
"I changed the record, clients still get the old value." No propagation — caches hold the old answer until TTL. Fix going forward: lower the TTL before a planned change; after, query the authoritative server directly to confirm, then blame caching for any lag.
"I created the name but it says it doesn't exist." Negative caching — the NXDOMAIN from before you created it is cached per the zone's SOA negative-TTL. Not a config error.
"The app resolves wrong but dig looks right." The app hit a different cache/resolver, an /etc/hosts entry, or the OS resolution order. Always note which resolver answered.
Identify which resolver answered and compare it to the authoritative answer. dig www.example.com shows what you got; dig example.com NS +short then dig @that-server www.example.com asks the source of truth directly, bypassing caches. Remember DNS needs both UDP/53 and TCP/53 — UDP for normal queries, TCP for large answers; a UDP-only firewall rule fails silently on big responses.
TLS gives you confidentiality, integrity, and server authentication — the client verifies the server's identity through a certificate that chains to a root it already trusts. Most "HTTPS is broken" reports are certificate or handshake problems, not encryption being "off."
"Certificate expired, but I'm sure it's valid." Check the client clock first. Validity is time-based, so NTP skew fakes "expired / not yet valid" on perfectly good certs.
"Works in the browser, fails in a script." Different trust stores — the OS, the browser, and each language runtime carry their own CA bundles; fixing one doesn't fix the others.
"Unknown issuer" on some clients only. A missing intermediate — the server must send the full chain; some clients fetch/cache intermediates, many don't. Wrong/missing SNI makes a multi-site server present the default cert, so you get a name mismatch even though the right cert exists.
The structure above is stable. But which TLS versions and ciphers are currently recommended or deprecated is a moving target — verify against current guidance rather than freezing a remembered list.
Pick a structure and run it the same way every time — ad-hoc poking is the novice signature. The single most efficient tactic is path bisection: test from progressively distant points and find where it breaks. Each rung that passes eliminates everything before it.
Misusing tools produces confident wrong conclusions. Know the exact scope of proof of each — especially the gap between what it shows and what people assume it shows.
| Tool | Proves | Does NOT prove |
|---|---|---|
ping | ICMP echo reachability + rough RTT | That a port is open, the app works, or MTU is OK |
traceroute / mtr | Forward-path hops, per-hop loss trend | The return path; a middle-hop spike is usually ICMP de-prioritization, not real latency |
dig / resolvectl | What a specific resolver returns | What the app resolves (it may use another cache/hosts file) |
ss / netstat | Local listeners & connection states | Whether a remote peer is reachable |
nc / curl -v | Whether a specific port/TLS/HTTP responds | General reachability beyond that test |
tcpdump / Wireshark | Ground truth of what's on the wire | Intent — you still interpret it; capture close to the problem |
iperf3 | Achievable throughput between two points | Latency- or app-bound problems |
The highest-value single distinction in the whole set is how a TCP port answers:
| You see | It means |
|---|---|
| SYN-ACK / "succeeded" | Port open, service listening |
| RST / "connection refused" | Host reachable, no listener on that port |
| timeout / no response | Filtered (firewall drop) or host down — ambiguous |
A fast RST is good news — it proves L3/L2 reachability and rules out a path/firewall drop; the service just isn't listening. A timeout is the ambiguous one. And a service bound to 127.0.0.1 is loopback-only (refused remotely); 0.0.0.0 is all interfaces — "works locally, refused remotely" means check the bind address first.
The gap between the mental model people carry and the real mechanism is where wrong root-cause conclusions come from. The highest-yield corrections:
| Belief | What actually happens | Troubleshooting signal |
|---|---|---|
| "NAT is a firewall" | NAT translates addresses; inbound is dropped only because there's no mapping — a side effect. The gatekeeper is the stateful conntrack table, a separate function. | Don't reason about exposure from "we have NAT." Reason from the actual ruleset/state. |
| "Ping works, so the network's fine" | Ping tests ICMP echo only. | Says nothing about ports, the app, MTU, or DNS. Keep going down the ladder. |
| "Connectivity is binary" | MTU problems are size-dependent: small packets pass, large ones drop. Connects, then hangs on bulk data. | SSH banner then freeze; page starts then stalls → PMTUD black hole (filtered ICMP type 3/4). Test with ping -M do -s. |
| "An L2 loop is just slow" | Ethernet frames have no TTL, so a looped broadcast multiplies forever. | Whole segment melts (100% switch CPU) right after a cabling/switch change → loop / STP failure. |
| "TIME_WAIT is a leak" | It's normal, on the endpoint that closed first, lasting 2×MSL to absorb stray duplicates. | Thousands on a server = it's closing connections (use keep-alive); can exhaust source ports on busy clients. |
| "More bandwidth fixes slowness" | A single flow's max ≈ window ÷ RTT. On a long-fat path the window (not the link) is the limit. | Single stream slow but parallel streams saturate → window/BDP, not capacity. |
| "Asymmetric routing is fine" | A stateful device (firewall/NAT) that sees only one direction drops the flow. | Connections that establish then stall in multi-path topologies. |
A firewall rule is match criteria → action. The classic match is the 5-tuple (source, destination, protocol, destination port) plus modern dimensions (zone, L7 app, user identity, device posture). Two rules of craft carry most of the value:
any in a permit's destination or service.Most rulesets obsess over inbound and leave outbound wide open. Reverse it: default-deny outbound, permit known destinations. Egress filtering constrains the post-compromise phase — exfiltration and command-and-control — that ingress rules never touch.
Drop vs reject is a real decision: drop (silent) slows a scanner but also your own troubleshooting; reject (RST / ICMP prohibited) gives fast, clear failure. Common practice: reject internally, drop at the internet edge. And know whether you're configuring a stateful device (permit the initiating direction; return is automatic — but asymmetric paths break it) or a stateless one (you must permit the return direction and the client's ephemeral range — the #1 stateless/NACL mistake).
Network location grants no trust; authorize per-resource and per-session on identity, device posture, and context, assuming the network is already breached. It's a strategy and architecture — not a product — decisions made at a policy decision point, enforced close to each resource. Adopt it staged: inventory → log-only to learn real dependencies → microsegment and tighten → move from IP-based to identity-based criteria.
The most common self-inflicted outage is severing your own access. Confirm a rule permitting your management access sits ahead of any new deny, use out-of-band access, and stage an auto-rollback (commit-confirmed / reload in) so a lockout self-heals. No rule makes a system "secure" — each mitigates a specific reachability against a specific threat and leaves residual risk; write and review rules in those terms.
Diagnosis is half the job; the other half is building and operating networks that are troubleshootable in the first place — the "sage advice" layer. The organizing principle: design for the person debugging it at 3 a.m. — usually future you. A small number of predictable, repeated patterns beats clever one-offs, because behaviour becomes inferable, faults isolate, and growth is mechanical rather than improvised.
Synchronized clocks are required for two things at once: cross-device log correlation (the basis of most investigations) and TLS certificate validation (§7). When clocks drift, your evidence and your certificates break together — and the cause looks like anything but time.
Everything above is one story. Here's what happens, in order, when you open https://www.example.com on office Wi-Fi — every troubleshooting failure mode lives at one of these steps, so "which step?" is always the useful question.
Bisecting the path (§8) is just walking this sequence until a step fails.
Scope it → "what changed?" → capture baseline → bisect the path (loopback → gateway → next hop → remote → port → TLS) → one hypothesis with a prediction → change one thing reversibly → document.
Follow the packet · problems live at boundaries · MACs are local, IPs are the journey · the mask drives local-vs-remote · authoritative is truth, the rest is cache · the capture doesn't lie.
SYN-ACK = open · RST = reachable, no listener (good news) · timeout = filtered or down (ambiguous). Ping proves L3 only.
DNS (which resolver vs authoritative; UDP and TCP/53) · MTU (small works / large hangs = PMTUD black hole) · a dropped link · a full table. APIPA (169.254) = no DHCP reached.
22 SSH · 53 DNS · 80 HTTP · 123 NTP · 443 HTTPS · 3389 RDP. Stable defaults; verify anything beyond the ubiquitous against IANA/vendor docs.
Accuracy over confidence — flag vendor defaults, version-specific and changeable values to verify, don't assert from memory. And nothing is "secure" in the abstract — name the threat each control addresses and the residual risk it leaves.
A teaching synthesis of the Networking Guide Set — it distills the mental models and highest-leverage mechanisms into one learning path; the deep-dives live in the source guides. For mechanism and reference see Subnetting & CIDR, IP Routing & Route Tables, DNS, TLS & Certificates, Networking Misconceptions / RCA, Network Sage-Advice, and Firewall & Network Rule Creation; for practice, the Diagnostic Command Card, Triage Decision Trees, and Self-Check Questions. Values that are vendor-specific, version-specific, or otherwise changeable (administrative-distance defaults, TLS/cipher currency, ports beyond the ubiquitous, ephemeral ranges) should be verified against current vendor/IANA/standards sources rather than taken from memory. Terms are in the Glossary; 00 — Start Here indexes the set.