Visibility answers what happened. Safety depends on whether the decisions behind it were sound, and right now almost no one is checking that.
“We have full observability” has become the answer to a question it doesn’t actually address. Ask an engineering leader how they know their AI systems are safe to give more autonomy, and this is what comes back: we can trace every request, replay every agent run, inspect every prompt. It’s said with real confidence, and the confidence isn’t unearned. The tooling is genuinely good now. It’s also answering the wrong question. Observability tells you what a system did. It says nothing about whether the system decided well, and safety was always about the decision, not the record of it.
I’ve written before about the shape of that gap: execution is observable, decisions are not, and the distance between the two widens as systems get more autonomous. This piece isn’t about the gap itself. It’s about what happens when an entire market starts treating a partial answer as a complete one, because that mistake has a well-documented history, and it didn’t happen in software first.
The recorder that didn’t prevent the crash
Photo: Ries Bosch on Unsplash
Commercial aviation solved its observability problem decades before software did. By the 1960s, flight data recorders and cockpit voice recorders, the “black box,” could reconstruct a flight down to the second: every control input, every instrument reading, every word spoken in the cockpit. It was a genuine leap, and it didn’t stop the crashes. Full execution data, and the accident rate didn’t move the way the industry expected it to.
The case that made this impossible to ignore was United Airlines Flight 173, which ran out of fuel and crashed short of the runway in Portland in 1978. The recorders worked exactly as designed: the landing gear light that wouldn’t confirm down, the captain’s decision to hold in a pattern and troubleshoot it, a flight engineer who mentioned the fuel state more than once without ever forcing the point. The record was complete. The crew still ran a functioning airplane out of fuel, in full view of instruments that recorded every reading correctly the whole time.
The industry’s response is the part worth sitting with. It didn’t add more sensors. It admitted the recorder had answered a question that wasn’t the one that mattered. What killed that flight wasn’t a hidden system failure. The record ruled that out immediately. It was a set of decisions: how the crew weighed a minor mechanical uncertainty against a hard fuel constraint, why the flight engineer didn’t escalate what he knew, why the captain fixated on one problem and lost track of a bigger one.
What followed was Crew Resource Management, a training discipline built out of NASA’s human-factors research into cockpit decision-making, with United among the first carriers to adopt it in the early 1980s. CRM didn’t touch the instrumentation. It targeted the layer the black box structurally couldn’t reach: how crews communicated uncertainty, challenged authority, and made calls under pressure. Aviation’s long decline in fatal accidents afterward is credited to several factors, but CRM is consistently named as one of the biggest, because it was the first serious attempt to instrument the decision, not just the execution.
The stage the AI industry is at
That history maps onto AI systems with almost no translation required. Modern observability platforms are, functionally, the black box: a complete, faithful record of what a system did, prompt by prompt, tool call by tool call, output by output. They’re a real achievement, the same way the flight recorder was. And teams are currently using that achievement the same way aviation briefly did in the years before Portland: as if having the recorder is the safety program, rather than the forensic layer underneath one.
The mechanism is identical, too. A trace can be perfectly clean and still describe a bad decision, because the trace only encodes the path a system took, not the alternatives it had, how strongly it favored the one it picked, or whether the context it weighted most heavily was the context that should have mattered. An agent that talks itself into the wrong tool order, that quietly favors a stale instruction over a current one, that produces a confident wrong answer, will generate an execution log that looks exactly as healthy as the one from a correct run. That’s not a tooling gap someone patches next quarter. It’s the same structural blind spot the 1978 investigators ran into: the recorder answers what happened, and the failure lived one layer up, in why it was chosen.
Why the false confidence is the expensive part
Here’s where this stops being a technical footnote and starts being a business problem.
Photo: Noah Gremmert on Unsplash
“We have full observability” is doing real work in rooms where AI risk gets approved: board updates, vendor due diligence, incident postmortems, the internal case for handing a system more autonomy. In every one of those rooms, it functions as an answer to “is this safe,” not “can we see it.” That substitution is quiet and it’s structural, because visibility and safety look similar from the outside right up until something breaks, exactly the way a clean cockpit trace looks identical whether the crew is making sound calls or slowly talking themselves into a bad one.
The consequence isn’t abstract. It shows up in three places, in order:
Deployment outpaces the thing that’s supposed to gate it. If leadership believes observability is the safety layer, the natural move is to extend more autonomy to these systems as fast as the tooling matures, because the box that needed checking got checked. That’s the same shape as aviation before Portland: instrumentation improved fast, and the industry (reasonably, in the moment) read that as safety improving at the same pace. It wasn’t. The gap between “we can see it” and “we understand it” doesn’t shrink on its own as systems get more capable; it widens, because more autonomous systems make more decisions per task, each one a place the trace won’t tell you anything about.
The vendor category is selling the wrong layer as the whole answer. An entire market has formed around AI observability in the last two years, and it’s being bought, correctly, as visibility tooling and marketed, less correctly, as risk mitigation. That’s not a knock on the tools; they do what they say. It’s a category-confusion problem, and category confusion is expensive precisely because it doesn’t surface until the thing it was supposed to prevent happens anyway. The parallel worth naming is the one security teams already lived through: a SOC 2 report or a clean audit log has never been proof that a system was secure, only that it was watched, and plenty of breached companies had both.
“We had full traces” won’t hold up as the defense it’s currently being treated as. As AI systems take actions with real financial, legal, or customer consequences, the question after an incident stops being “can you show us what happened” and becomes “can you show us the decision was sound, or that you’d have caught it if it wasn’t.” A complete execution trace answers the first question well and the second one not at all, and the gap between those two answers is exactly where liability, regulatory exposure, and lost trust are going to concentrate as this decade plays out.
What this means for where the market goes
The structural position that ends up mattering isn’t “who has the best dashboard.” It’s the same lesson aviation learned the hard way: the instrumentation layer becomes table stakes fast, and the durable advantage sits one layer up, with whoever can show a decision was actually sound rather than just show that it happened. Teams and vendors still selling visibility as the finished story are going to look, in a few years, the way pre-CRM aviation looks in hindsight: not wrong that the recorder mattered, just early on believing it was the whole job.
That’s the direction I’m building Nalyqor toward: helping teams understand not just what their AI systems did, but whether the decisions behind them were actually sound. Over time, that’s the question customers, auditors, and regulators will increasingly expect organizations to answer.