![Model Citizen](https://www.model-citizen.org/brand/logo-color.png)

# Data Incident Runbook Template

*Model Citizen — a step-by-step resolution guide for when a pipeline, dashboard, or dataset breaks in production. Copy this file per-incident-type (e.g., "Nightly ETL failure," "Dashboard data gap") and fill in the specifics ahead of time — the goal is that on-call follows a proven playbook instead of troubleshooting cold.*

---

## 1. Severity levels

| Level | Definition | Response time |
|---|---|---|
| **SEV-1** | Executive-facing report or revenue-impacting pipeline is wrong or down. | Immediate — page on-call. |
| **SEV-2** | Internal dashboard or non-critical pipeline is wrong or delayed. | Same business day. |
| **SEV-3** | Cosmetic issue, single non-critical table, no downstream impact yet. | Next business day. |

## 2. Roles

| Role | Responsibility |
|---|---|
| **Incident Commander (IC)** | Owns the response end-to-end. Doesn't have to fix it themselves — makes sure someone is. |
| **On-call engineer** | Triages, diagnoses, and applies the fix or mitigation. |
| **Communicator** | Posts status updates to stakeholders on a fixed cadence (see §5). Can be the same person as the IC for SEV-2/3. |

## 3. Triage checklist (first 15 minutes)

- [ ] Confirm the incident is real — reproduce it, don't take a single Slack message as ground truth.
- [ ] Assign severity (§1) and an Incident Commander.
- [ ] Check the obvious first: did an upstream source change schema, did a scheduled job fail silently, did credentials expire?
- [ ] Identify blast radius — which dashboards, reports, or downstream tables consume the broken data?
- [ ] Post the first status update (§5), even if it's just "we're aware and investigating."

## 4. Diagnosis → Fix

1. **Isolate**: which stage broke — ingestion, transformation, or the serving layer? Check the last-known-good run.
2. **Contain**: if the bad data has already propagated, decide whether to pause downstream consumers or let it ride with a visible caveat.
3. **Root cause**: identify the *why*, not just the *what* — a fixed value today that recurs next week isn't fixed.
4. **Fix or mitigate**: apply the smallest change that resolves the immediate issue. Larger structural fixes go in §7, not into the incident itself.
5. **Verify**: confirm downstream reports/dashboards now reflect correct data before declaring resolved.

## 5. Communication cadence

- SEV-1: update every 30 minutes until resolved, even if the update is "still investigating."
- SEV-2: update at start, at any major status change, and at resolution.
- SEV-3: update at resolution only.

**Update template:**
> **[SEV-X] [System name] — [status: Investigating / Identified / Monitoring / Resolved]**
> Impact: [who/what is affected]
> Current understanding: [one sentence]
> Next update by: [time]

## 6. Resolution

- [ ] Confirm the fix in production, not just in a test environment.
- [ ] Notify affected stakeholders directly (not just in the incident channel) that the issue is resolved.
- [ ] Log the incident timeline (detected → identified → resolved) for the postmortem.

## 7. Postmortem (within 3 business days)

- **What happened** — plain-language summary, no jargon.
- **Timeline** — detection, diagnosis, fix, verification, each timestamped.
- **Root cause** — the actual mechanism, not "human error."
- **What made this hard to catch** — the monitoring or alerting gap that let it reach production.
- **Action items** — each with an owner and a date, not a vague "we'll look into it." Distinguish between the *incident fix* (already done) and the *systemic fix* (prevents recurrence).

---

Built by [Model Citizen](https://www.model-citizen.org) — real incident-response structure, not just a template for its own sake.