Death by a Thousand Tokens

It's an easy trap to fall into: wire an agent to pull a metrics API every morning, reshape the JSON, and drop the result into a table. It works on day one.

Then the bill arrives, and it's for something dumb. A few thousand tokens spent every morning asking a language model to reformat data it never needed an opinion on. Worse, it occasionally gets the reformatting wrong, because a probabilistic model is doing work a five-line script would do identically every time. And every run adds another batch of stale JSON to the context window — which is exactly where the model's actual reasoning happens.

That trap catches almost everyone right now, because a model looks like a universal runtime: if you can describe a task in a sentence, you can prompt it. But a model is a reasoning engine you rent by the token. Point it at a cron job or hand it your data store and you're paying a premium for work that never needed a brain in the first place. The skill worth learning isn't prompting better. It's knowing what to keep away from the model entirely.

The pattern shows up in small ways too — someone uploads a screenshot to a chat model and asks it to retype the text, burning tokens on a job a free screenshot tool already does in two seconds. Trivial on its own. Run that shape of mistake a few thousand times a day across a team, and it's a real line on the invoice.

Sort the work before you decide where it runs

Every task an agent touches falls into one of two buckets. Judgment goes to the agent: drafting, summarizing, classifying, deciding whether something actually matters, handling ambiguity. Everything else goes to the app around it: scheduled and recurring jobs, API calls and data transforms, storing and querying data, anything that has to come out exact and repeatable on every run.

Determinism isn't a compromise here — it's the feature you actually want. A job that runs every morning should be boring.

Here's the part worth building the whole architecture around: you spend tokens once, building the machine. Then the machine runs for free.

The mistake everyone makes: memory as a database

There's a specific version of this worth calling out on its own, because it's the one I see even careful teams walk into. They let the agent's memory or notes become the database. Structured records — metrics, pipeline stages, customer lists — get dropped into a note because the note is right there and the agent already reads it.

Notes make a bad database for the same reason an ungoverned spreadsheet does: no schema to enforce consistency, no query, so the agent has to re-read the whole note into context just to pull one field, and the whole thing degrades as it grows. It's the same argument I keep making about semantic layers, just one level up the stack: structured, high-volume, queryable data belongs in a real, governed store. Unstructured, interpretive context — preferences, tone, the reasoning behind a past decision — is what notes are actually for. Match the shape of the data to the tool, and both problems disappear at once.

Build the machine once, then let it run

Take a recurring workflow — flagging at-risk customers before they churn. Design the schema and build the pipeline once; that's where the tokens go. Automate the mechanical part — pulling the latest status, updating a table — as a scheduled job that touches no model and costs nothing per run. Then reserve the model for the one thing it's actually there for: once a week, querying the flagged accounts, cross-referencing why they're slipping, and writing the summary a human reads.

Build that same workflow entirely inside the model instead — status checks, queries, and all — and you pay tokens on every row, every run, forever, for work that was never a judgment call to begin with.

Most of the tools built for this moment miss that split. Chat-first agent platforms have no app layer to hand work off to, so every scheduled task and lookup runs as tokens, because tokens are all they have. No-code app builders have the opposite gap: deterministic execution all day, with nothing deciding what actually deserves attention. The architecture that survives cost pressure treats the agent and the app as two different tools doing the same job, not one tool pretending to be both.

What to keep off the model

Before the next prompt, run the task through a short list. Does this need judgment, or is it mechanical? Does it repeat on a schedule? Is the agent making an API call it could hand off to code instead? Is structured data sitting in a note rather than a database? Is data in the context window to be reasoned over, or just stored?

None of this is an argument for less AI. It's an argument for spending intelligence exactly where it earns its cost, and letting cheap, deterministic code carry everything else. Build the machine once. Then stop paying to run it.