![Model Citizen](https://www.model-citizen.org/brand/logo-color.png)

# LLM Semantic Grounding Checklist

*Model Citizen — an assessment checklist for auditing whether your data models are safe to hand to an LLM or generative AI agent. If an agent is going to answer "what's our revenue?" it needs to inherit your organization's actual definition of revenue, not infer one from raw tables. Work through each section before connecting an agent to production data.*

---

## 1. Definitions are centralized, not tribal knowledge

- [ ] Every business metric an agent might be asked about ("revenue," "active user," "churn") has exactly one authoritative definition, not one per team or dashboard.
- [ ] That definition lives in a semantic layer or metrics layer (dbt semantic layer, LookML, a documented view) — not scattered across ad-hoc SQL in five different notebooks.
- [ ] The definition is machine-readable, not just written in a wiki page a human has to translate into SQL by hand.

## 2. The agent reads the definition, not the raw tables

- [ ] The agent's data access is scoped to the semantic layer / metrics layer, not direct query access to raw source tables.
- [ ] If the agent generates SQL, it's validated or constrained against approved models — not free-form querying of anything in the warehouse.
- [ ] Ambiguous business terms ("this quarter," "active," "customer") resolve to one place the agent is grounded in, not left to the model's own judgment.

## 3. Access controls extend to the agent, not just human users

- [ ] Row-level and column-level security that applies to human analysts also applies to the agent's queries — an agent isn't a backdoor around permissions.
- [ ] PII and sensitive fields are masked or excluded from what the agent can retrieve, by default, not by relying on the model to "choose" not to surface them.
- [ ] There's an audit log of what the agent queried and returned, same as you'd want for a human analyst with broad access.

## 4. The model is evaluated against known-answer questions

- [ ] You have a test set of business questions with known-correct answers, and you run the agent against it before and after any change to the underlying models.
- [ ] Failure modes are captured — not just "did it get the number right," but "did it use the right definition to get there."
- [ ] There's a process for what happens when the agent is confidently wrong, not just when it says "I don't know."

## 5. Guardrails for what the agent won't do

- [ ] The agent has explicit boundaries on what it can act on (read-only vs. write access, if any) — grounding solves *accuracy*, not *authorization*.
- [ ] There's a fallback behavior for questions outside its grounded scope ("I don't have a definition for that metric" beats a fabricated one).
- [ ] Someone owns monitoring the agent's outputs in production on an ongoing basis — this isn't a one-time setup task.

## 6. Change management

- [ ] When a metric definition changes upstream, there's a process that updates what the agent is grounded in — not a silent drift between the semantic layer and what the agent was trained/prompted on.
- [ ] New metrics or data sources go through the same grounding review before the agent can be asked about them.

---

**Scoring guide:** if you checked every box in sections 1–3, you have the prerequisites for a grounded agent. Sections 4–6 are what keep it grounded once it's live — skipping those is how a demo that worked in week one starts hallucinating by week eight.

Built by [Model Citizen](https://www.model-citizen.org).