← All Writing

Editorials

Why Every AI Incident Becomes an Engineering Incident

Sales ops owns the workflow. Engineering owns the incident. The gap between those two sentences is more expensive than the hours it burns.

A discount-approval agent sits inside a sales workflow. Reps submit exception requests, the agent approves the ones that fall inside policy and escalates the rest, and for two quarters it does that well enough that nobody thinks about it. Then finance notices that approvals in one segment have drifted looser than the quarter before, several points looser, with no policy change to explain it.

That is a sales operations problem. The workflow belongs to sales ops. The policy belongs to finance. The number that moved is a revenue number.

Watch where the question goes anyway.

It goes to engineering. More precisely it goes to the two or three engineers who have touched that agent, it arrives as a Slack thread with a screenshot in it, and it stays open for most of a week.

Nobody decided this. It isn’t in anyone’s job description, and if you asked the VP of Engineering whether her team owns discount policy she would say no, obviously not. But the org chart and the escalation path disagree about who owns this question, and in practice, that escalation path usually wins.

The question routes to the evidence, not to the owner

The reason is unglamorous. The only artifact that touches the question lives in a system that only engineers can read.

Sales ops has the outcome: approval rates moved. Finance has the policy the rates are supposed to reflect. Neither has anything that connects one to the other. What sits in between is a set of agent runs, and those live in traces, logs, and a tool-call history that requires knowing what the agent was wired to do in the first place.

So the question gets handed to whoever can pick up the evidence. I’ve argued before that ownership follows evidence, which sounds like a governance abstraction until you watch it happen on a Tuesday. The org chart assigns the workflow. The evidence assigns the incident. When they disagree, the evidence wins, because it is the only concrete artifact anyone can actually interrogate.

That’s how a revenue question becomes an engineering ticket without a single person choosing to route it there.

What engineering can actually answer

To be answerable at all, the question has to change shape on the way in. “Has the agent been approving things it shouldn’t?” is a judgment question. It becomes “what do the runs show,” because that is the only version of it the available evidence can address.

And the runs answer. Every approval fell inside the bounds as encoded. No errors. Tool calls returned cleanly. Nothing retried, nothing timed out, no anomalies in the sequence. An engineer spends three days establishing this, and the finding is real: the system did what it was built to do, on every one of those approvals.

Then that answer goes back to sales ops, who are exactly where they started, now holding a screenshot of a trace.

The expensive part of this is not the three days. It’s that the organization spent its least schedulable resource producing a technically accurate answer to a question nobody asked. The question was whether the judgment was sound. The artifact records execution. Those are different objects, and no amount of engineering time converts one into the other.

The bill a VP of Engineering actually pays

Hours are the easiest thing to count here and the least interesting.

What makes this expensive is that the work is unschedulable. It arrives on finance’s timeline, not engineering’s, and it arrives as a question that can’t be scoped in advance because nobody knows yet whether the answer takes an afternoon or a week. You cannot put it in a sprint. You can only absorb it, which means something else moves.

It also concentrates. It doesn’t land on “engineering,” it lands on the small number of people who have enough history with that specific agent to interpret its runs at all. I’ve written about why that group stays small and why hiring doesn’t widen it quickly: most of what makes them able to answer is accumulated familiarity with one system, and that isn’t transferable in an onboarding doc. So the same two names appear on every one of these, regardless of headcount, regardless of rotation.

And it recurs, because nothing survives it. The investigation produces a conclusion, someone adjusts a prompt or tightens a bound, and the reasoning that got them there evaporates into a closed Slack thread. Six weeks later a similar drift appears in a different segment and the same two people start over from the runs, at the same rate, for the same week.

None of this shows up as a line item. That’s what makes it hard to manage against. It shows up as an engineering org that keeps missing commitments by a margin nobody can quite account for, which is a considerably worse way to learn about a cost than seeing it on a budget.

The ceiling nobody put on a roadmap

The part I think gets missed at the executive level is what this does to the company’s appetite for AI.

Every new agent a company deploys adds to the same queue, held by the same small group. The second agent doubles the surface. The fifth one doesn’t quintuple it exactly, but it doesn’t spread the load either, because the interpretive work still concentrates on whoever knows each system’s history. Meanwhile the failures that generate these questions are the quiet kind, the ones where nothing errors and the output looks fine, so they surface late and land as surprises from finance, support, or a customer.

Which means one of the practical constraints on how much AI a company can run is how much unexplained behavior the engineering organization can absorb before something it was supposed to ship stops shipping. Not model capability, and usually not budget.

Nobody writes that constraint down, so the rationing happens informally, and it resolves one of two ways. Some teams slow down: the next workflow doesn’t get automated, and the reason given in the meeting is priorities, not that the last one costs three engineer-days a month to keep understood. Other teams keep deploying and quietly stop asking. The agents run, the logs say they ran, and no one reviews the decisions inside them until a customer or an auditor does it for free.

The second option is more common, because it costs nothing this quarter.

What would actually change the routing

It’s tempting to treat this as an incident-management problem, and it isn’t. A better process routes the question faster and to the right people sooner. It does not change what the artifact at the end of the route is able to answer, and that limit is the whole reason the loop is expensive.

The thing that would change the routing is a different artifact. If the evidence about a decision existed in a form the workflow owner could read, sales ops could answer a sales ops question without borrowing three days of engineering to translate it. Not because engineers stop being useful, but because the incident would stop defaulting to them for the reason it currently does: that they were the only people who could interrogate the evidence at all.

Ownership would follow the evidence back to where the org chart wanted it in the first place.

That’s the direction I’m building Nalyqor toward. The bet is that these questions cost what they do less because of how incidents get handled than because of what the system failed to retain about the decision.

Fix the artifact, and a surprising amount of the routing problem changes with it.

What I’m confident about is the diagnosis. Right now, in most companies running agents in production, a question about business judgment gets answered by the most expensive people available, in a register the person who asked it can’t use, and then gets asked again in six weeks.

If you're navigating how AI systems make decisions in your own product, this is the kind of problem I advise on.

Book a call →