Observability
Knowing what your system is actually doing, before users tell you.
AI can write the code; it can't tell you checkout is silently failing for 3% of users on one browser. The durable skill is knowing what to measure and reading the signal under pressure. When it's on fire is the worst time to learn it.
Ready to test yourself on this kind of call?
Practice debugging under pressure →What you own
- ▪Deciding what's worth measuring (the targets/SLOs)
- ▪Judging signal vs noise during an incident
- ▪The root-cause call
- ▪What is actually user-facing
▸Hand to AI (4)
- Adding structured logging + trace context to handlers
- Writing alert rules from a described target
- Drafting dashboards
- Summarising a noisy log dump
What to learn (the durable stuff)
The three pillars
Logs say what happened. Metrics say how much and how often. Traces follow the path of one request.
Symptoms vs causes
An error spike is the symptom. The trace tells you the cause.
Percentiles over averages
p95 and p99 latency (the slowest 5% and 1% of requests) is what users actually feel. The average hides it.
What to alert on
Alert on user-facing symptoms like error rate and latency, not every internal blip. Alert fatigue kills response.
Trace IDs and structured logs
Tag every log line with the request's id so you can follow one request across services.
Sampling and cost
Tracing every single request is expensive. Sample smartly.
Current tools (these change fast)
Practice this scenario
Your app got 20x slower after launch but CPU looks fine. You have request traces and DB query logs. How do you find the bottleneck, and what's the one graph you'd pull up first?
Practice debugging under pressure →