The models got good enough that small teams stopped needing an ML engineer. Then the job came back, in a shape nobody can hire for.
A lot of the AI running in production right now isn’t being run by AI teams. It’s being run by a support lead who wired an agent into the ticket queue, a founder who put a model in front of the approval flow, an ops manager whose “automation” now decides which refunds go through. Five people, sometimes two, one of whom writes code. No data scientist. No ML engineer. Nobody with the word “model” in their title.
None of that is a failure of ambition. It’s what the last couple of years of model progress made possible, and small teams did the rational thing with it: they shipped. The API took away the part of the work that used to need a specialist. What it didn’t take away was the job. That moved, onto whoever owns the workflow, and it tends to show up at the worst possible time.
The job that left, and the one that arrived
The ML engineer small teams “don’t need anymore” was a real role with a real definition. Pick the model, prepare the data, train or fine-tune, tune the thresholds, ship the artifact, retrain when it drifts. Foundation models removed the requirement to do most of that before a team could ship AI at all. A support team no longer needs to train a model before putting one into a workflow. They can prompt a hosted model, give it tools, connect it to production, and go.
What nobody wrote a job description for is the role that opened up in its place. When the agent approves something it should have escalated, someone has to figure out why. When a workflow that ran fine for six weeks starts producing answers that are structurally correct and quietly wrong, someone has to notice, and then explain it to a person who doesn’t read traces. When the monthly bill doubles and every individual call looks reasonable, someone has to work out which behavior changed.
That is not a modeling job. It is a judgment job: knowing what normal looks like for this agent, on this workflow, and being able to say whether a given decision was sound and, if it wasn’t, what changed. On a team with an ML engineer, that person absorbs it without anyone deciding they should. On a team without one, it lands on the workflow owner, and it lands the way incidents do, at 11pm, with the logs open and no idea what they’re looking for.
I’ve written about the gap at the category level before, execution is observable and decisions are not. This piece is about who actually pays for it. Mostly it’s the smallest teams, and I think that’s why the gap gets filled from the bottom of the market up rather than the top down.
Why “just hire one” doesn’t work
The obvious response is to staff for it. Hire the ML engineer. I’ve watched a few small teams try, and it rarely works, and money is the least interesting reason why.
Start with what the salary buys. ML engineers are expensive because the thing they’re scarce at is building models. A team running an agent off a hosted API needs something else from them: behavioral diagnosis, production judgment, the ability to look at a run and say whether it was normal. That’s a real skill, and some ML engineers have it, but it isn’t the part the salary is paying for. You’d be hiring a surgeon to take a temperature.
Then there’s the shape of the work. A small team doesn’t need a full-time person to explain agent decisions. It needs one urgently when something strange happens, and then perhaps not again for weeks. That’s on-call, not a role, and small teams already know how filling an on-call gap with a headcount goes: the person gets bored and leaves, or gets pulled onto product work and is busy exactly when the incident hits.
The hardest one, and the one I think most teams miss, is that the job needs context no new hire has. Judging whether an agent’s decision was sound depends on knowing what that agent normally does. Its typical tool sequence on this task, its usual confidence, the fact that it always retries the CRM lookup twice and that’s fine but three times means the integration is degrading. That’s history, not skill. A senior ML engineer walking in on day one doesn’t have it any more than the support lead does. The person who has it, roughly, is the workflow owner, and they have no way to use it, because nothing is writing it down. Every investigation starts from zero. The team isn’t learning how the system behaves so much as rediscovering it, each time, from the logs.
So the job stays unassigned. Sometimes the founder or the one engineer picks it up, late, at the cost of whatever they were supposed to be building that week. Sometimes a senior engineer becomes the de facto owner of every AI incident because they’re the only one who can read the trace (that’s an organizational-cost story I’ll come back to separately). Most often, nobody picks it up. The workflow runs, the logs say it ran, and the decisions inside it are never reviewed by anyone.
What “nobody does it” actually costs
The failures that follow are silent, which is why small teams underweight them. A refund agent that starts approving a category it used to escalate doesn’t throw an error, it just approves. An assistant that starts favoring a stale instruction over the current one produces confident, well-formatted, wrong output, and the trace of that run looks like the trace of a good one. I’ve written about the anatomy of those failures; the point here is that a small team has nobody positioned to see them, so the first signal is usually a customer, an auditor, or the invoice.
The invoice is the one place the problem becomes visible without anyone looking for it. When a small team’s model bill climbs and nobody can say why, the money is real, but the more useful way to read it is as evidence. Wasted spend is what unreviewed decisions look like from the outside: retry loops nobody caught, context nobody trimmed, an expensive model doing work a cheap one used to do because a routing decision drifted. It’s the hidden AI tax I’ve written about, showing up on a five-person team where nobody was watching the decisions in the first place.
The role that becomes a layer
Small teams have been here before. Two decades ago, running software in production meant having someone who could rack a server, patch an OS, and babysit a database at 3am. Small companies couldn’t afford that person either, so the founder did it, usually at 3am, usually badly. Small teams never did learn to hire sysadmins. Instead the role became a layer: managed infrastructure and managed databases absorbed that judgment into a service a two-person team could switch on, and the expertise moved into the platform, where it could serve thousands of teams instead of one.
The same absorption is coming for the ML engineer’s judgment job, and I expect it to come for small teams first, because they feel the absence most acutely and have the least ability to paper over it with headcount. A large enterprise can hire its way to a partial answer. A five-person team running an agent in production can’t, and there are far more of them.
That’s what makes the small team the entry point for this whole category rather than an afterthought to it. Enterprise governance and drift reporting are where the story expands. But the wedge, the place a decision layer first earns its keep, is a workflow owner with no ML engineer asking a question they can’t currently get answered: why did it do that?
What this means for the market
A couple of things follow from that, and both point away from where most vendors are aimed.
The teams gaining the easiest access to production AI are often the teams least equipped to explain what it does once it gets there, and the tooling market has been built for the opposite customer. Observability platforms assume an engineer who reads traces. Eval frameworks assume someone who writes them. Neither is wrong, but neither serves the person actually holding the pager on a small team, who needs an answer in plain English, not another surface to interpret.
And whoever ends up filling that role for small teams isn’t really selling a feature. They’re selling the always-on version of a hire the team couldn’t make: the ML engineer’s production judgment, on every run, without the ML engineer. I know that sounds like a pitch. It’s closer to a description of the job’s shape: intermittent, history-dependent, translation-heavy, and it has to run continuously. Work like that doesn’t fit a person. It fits a layer.
That’s the sysadmin story again, and the teams that switch the layer on first will be the ones who could never have hired the person. I’m building Nalyqor on the belief that the decision engineer small teams can’t hire is the one they end up switching on, and that the market for the whole decision layer gets built from there.