What changes when an LLM drives the flow
There's a moment, somewhere in the first months of building a production agent, when you realize the job isn't the job you trained for.
I build patient communication systems. Agents that call people, confirm appointments, answer questions, escalate when something sounds wrong. The stack is familiar. The integrations are familiar. What isn't familiar is that the fundamental contract of software development has inverted, and most of the habits I brought with me were built on the old contract.
Here's the inversion in one sentence. In traditional software, if there's no requirement, nothing happens. In an agentic system, if there's no requirement, something happens anyway.
Everything else follows from that.
The default flipped from deny to allow
Think about how product knowledge used to get into a codebase. You'd sit with the client and ask what should happen if the patient says they're not sure about the time. They'd answer. You'd write a branch. The ambiguity got resolved at implementation time, by you, and then it stayed resolved. The knowledge was compiled into control flow.
If you forgot to ask, the system did nothing. A 404, an unhandled state, a crash. Annoying, but visible and inert.
An agent fails the other way around. The ambiguity you never resolved shows up at runtime, and the model resolves it for you. Plausibly, confidently, and slightly differently each time. Nobody gets an error. A patient gets an answer that nobody at your company ever approved.
Traditional code starts with an empty capability set that you add to. An agent starts with a huge one that you carve down. It's closer to writing a whitelist than writing features, and it means the absence of a requirement is no longer safe.
There's a practical consequence people miss here. Refusal is now something you build. "I can't help with that," escalation, the graceful stop. Those used to be free, because they were what happened when you hadn't written anything yet. Now they're deliberate work.
Three layers of control, and only one of them holds
Once you accept that you're carving down instead of building up, the question is what you carve with.
There are three levers. Almost every team reaches for the weakest one first, because it's the fastest to apply.
The first is capability. What the agent structurally cannot do. Tools it doesn't have, permissions it lacks, data it can't reach. This one is absolute. A model can't misuse a tool that isn't in its context.
The second is validation. What your code checks, rejects, or enforces after the model proposes something. Output validators, rate limits, iteration caps, deterministic state writes. Reliable, because code doesn't have moods.
The third is instruction. What the system prompt asks for. Real and useful, and also the one that decays.
It decays in a way you can actually measure. At turn two your system prompt might be 40% of the context. At turn twenty five it's 3%, buried under thousands of tokens of conversation that are more recent and more vivid. The rule doesn't disappear, it just gets outweighed. And once the agent has broken it a single time, that violation sits in its own context as an example of how it behaves. It starts following itself.
Which is where I ended up with a rule of thumb that settles a surprising number of design arguments. Prompts express preferences. Code enforces limits.
If something must never happen, it can't live in a layer that erodes with conversation length. So for every requirement, ask which of the three it belongs to. Most of what ends up in prompts belongs one layer down.
Correctness became statistical
The second thing that breaks is your definition of working.
It's tempting to blame non-determinism, but that isn't really it. Even at temperature zero you'd have this problem, because the input space is unbounded. Patients say things nobody enumerated. Traditional testing works by covering a finite set of paths, and here the paths aren't finite.
So correctness stops being binary. You stop asking whether it works and start asking what percentage, on which distribution, and whether that number is moving.
Three things follow.
Your eval suite is the only instrument you have. Not a nice to have, the actual measuring device. It works in three layers. Unit evals on discrete steps, checking whether it picked the right tool and pulled the right parameter. Trajectory evals on whole runs, checking whether it reached the goal regardless of the path it took. And continuous sampling of production traces, because the distribution shifts underneath you whether or not you're watching.
The percentage also means very little without the distribution attached. 94% on last quarter's traffic tells you almost nothing about this quarter if the caller mix changed. The dataset has to keep getting refreshed from real traces, otherwise the number stays reassuringly stable while quietly measuring a different world.
And not all failures are the same size. Averaging them into one score is how you ship something dangerous with a healthy dashboard. What works better is one quality rate you optimize, plus a small set of safety conditions where the target is zero and any occurrence is an incident. Those don't get managed as percentages. They get enforced in code, because in healthcare "99.9% of the time we don't do the dangerous thing" isn't a sentence anyone should be comfortable saying out loud.
Judging the things you can't check
Some of what matters can't be checked deterministically. Was the tone right for someone who just said something worrying. Was the response technically correct but useless.
The usual answer is to have a model score it, and that works. But a judge is a classifier, and an unvalidated classifier is just a confident opinion. Label a couple hundred examples by hand. Then measure agreement with Cohen's kappa rather than raw accuracy. If 90% of your calls are fine, a judge that says "fine" every single time scores 90% accuracy and carries no information at all. Kappa reports that honestly as zero.
Use binary or three-point scales with written criteria, not one to ten. Nobody has a stable definition of what separates a six from a seven, so the scores cluster around seven and a half and the differences between versions are mostly noise.
And have two humans label an overlapping slice. If they don't agree with each other, your rubric is ambiguous and no judge is going to rescue it.
Your existing observability stack is blind
An agent handles a call badly. You open Datadog. HTTP 200, normal latency, no errors.
APM was built for a world where returning a 200 quickly and doing the right thing meant roughly the same thing. It sees the envelope. It can't see the reasoning tree, and an agent run is a tree. Model call, tool call, result, model call, subagent, back up again.
So even the mechanically checkable failures stay invisible. The agent called the same lookup eleven times with identical arguments. The context truncated silently and it answered from half the data. A tool returned an error string that the model read as content. None of that surfaces as an error anywhere.
What you need instead is instrumentation that understands agents. Spans carrying prompts, tool arguments, tool results, token counts, model versions, and the parent-child structure connecting them. If there's PHI involved, redaction happens at instrumentation time, not at query time.
The vendor choice matters less than people think. Use OpenTelemetry's generative AI semantic conventions and you can move between Langfuse, LangSmith, Phoenix, or whatever exists in eighteen months without reinstrumenting everything. The schema is the durable decision.
A debugging habit worth building
When you see repeated identical tool calls, the instinct is to add a prompt instruction telling the agent to stop doing that. Resist it.
A model repeating a call almost always means the tool result didn't give it what it needed and it had no other move. The tool returned an empty object for both "no results" and "error." Or it returned something ambiguous. The root cause is the tool's return contract, not the prompt.
So fix it at the tool. Return an explicit status the model can act on, and make errors instructive. Something like {"status": "not_found", "next": "Ask the caller to confirm their date of birth."} gives the model somewhere to go. An empty object gives it nothing but retry.
Then add the iteration cap anyway, in code, where it can't be talked out of it.
Context is a resource you manage
Thirty turns into a conversation the window fills up. You have four options and only three of them are real.
You can truncate, dropping the oldest turns. This means the agent forgets what the caller asked for at the start while remembering small talk from a minute ago. It's almost never the right deliberate choice.
You can compact, summarizing older turns and keeping recent ones verbatim. That's the default for continuous conversation, with one hard rule attached. Decisions and commitments survive word for word.
You can offload to structured state, keeping facts in an object outside the window and reinjecting them each turn.
Or you can spawn a subagent, giving a self-contained subtask its own fresh context so it burns tokens internally and returns a small result.
The reason compaction needs that hard rule is that summarization is lossy in a biased direction. A summarizer optimizes for narrative coherence, so it keeps the gist and drops the specifics, which is exactly the wrong half. "Patient said they have not taken it yet" becomes "discussed medication," and the negation is gone without a trace. Qualifiers get stripped, so "maybe Thursday, but I need to check with my daughter" becomes "confirmed Thursday." Provenance disappears entirely, and afterwards the agent can't tell what the patient actually said from what it inferred.
Hence the split. Compaction is for conversation. State is for facts.
Anything operationally load-bearing shouldn't be sitting in transcript text at all. Then compaction can only damage the chat history, never the things you can't afford to lose.
Writing to state brings its own failures, and they're worth naming. Premature writes, where a tentative "probably Thursday" gets recorded as confirmed and nothing ever reexamines it. Missed updates, where the patient changes their mind at turn twenty two, the conversation reflects it and the state doesn't, and state wins because state is what gets reinjected. Silent overwrites with no history, so you can't reconstruct what happened.
The mitigations have the same shape as everything else here. Write values with a confidence level and require an explicit step to promote them. Append with timestamps instead of mutating in place, so revisions stay auditable. And where a fact can be determined deterministically, like the caller answering a verification question correctly, let code set it. The model proposes, code commits.
Tool definitions are prompts
This one took me embarrassingly long to internalize. A tool definition isn't an API spec. It's a prompt, read by someone with no documentation, no examples, and exactly one attempt.
Four things decide whether it's good.
Say when not to use it. Models over-call tools, and most selection errors come from two tools whose descriptions sound interchangeable. "Use this to find open slots. Do not use it to confirm a booking."
Constrain the parameters. Enums instead of free strings. A parameter typed as query: string is an invitation to invent structure.
Make errors instructive, for the reason above.
Keep payloads small. Returning 4,000 tokens to convey four available slots isn't a performance problem, it's a context engineering bug. Every unnecessary token is noise the model has to look past.
At scale the problem isn't the number of tools, it's semantic overlap. Twenty five well-separated tools work better than eight that blur into each other. If you have get_patient, lookup_patient_record, and fetch_patient_details, the model is guessing, and so would a human reading only those names.
The fixes are consolidation, phase scoping so you only expose scheduling tools during scheduling, or routing to subagents with small coherent toolsets. And you make the whole thing measurable rather than a design review argument by building a tool selection eval. Real requests, the tool that should have been called, a number that moves when you fix something.
The part nobody warns you about
Here's the shift I didn't see coming. You need more product knowledge than before, not less.
In a traditional project, product knowledge got captured once, encoded into branches, and stayed put. With an agent, every gap in domain understanding becomes a place where the model invents policy at runtime.
So the tacit judgment the client's staff apply without thinking, the twenty edge cases a day they resolve through experience nobody wrote down, has to become explicit text. The model has no clinic experience to fall back on.
Two things follow, and both changed how I work.
The eval dataset is product knowledge in executable form. "This scenario should produce that outcome" is a requirements document that fails loudly when reality diverges from it. It's probably the most durable artifact the team produces, more durable than the prompts, which get rewritten every time you change models.
And subject matter experts move upstream and stay there. Someone who genuinely understands patient communication needs to be labeling and reviewing continuously, not approving a spec at kickoff and disappearing. That's an ongoing role, not a phase.
The uncomfortable version of all this is that the model exposes how much of your product spec was never actually specified. You will discover, sometimes in front of the client, that there was no agreed answer to half of it.
What I'd tell myself eighteen months ago
Non-determinism isn't the hard part. The hard part is that you've moved from a system that does only what you built to a system that does anything unless you prevent it, and your instincts, your testing habits, and your monitoring stack were all built for the first kind.
Better observability tells you when the probabilistic part failed. Shrinking the probabilistic surface means it fails less often. Most teams invest heavily in the first and barely at all in the second.
You need both.