Your autonomous agents are flying without a black box, and aviation already solved this
On 27 March 1977, two Boeing 747s collided on a single runway at Los Rodeos airport in Tenerife. Fog had rolled in off the hills. 583 people died, which still makes it the deadliest accident in aviation history. The KLM jet started its takeoff roll while the Pan Am jet was still taxiing towards it through the murk, looking for the exit it had already missed.
Everyone on the KLM aircraft died, including the captain. He was Jacob Veldhuyzen van Zanten, head of the airline’s 747 training programme and one of the most senior pilots KLM had. No one from his flight deck survived to explain why he began rolling without takeoff clearance. The recorders were the only witnesses left.
Your autonomous agents are in roughly the position van Zanten was in: trusted and capable, acting on their own read of a situation that nobody’s double-checking in real time. When one of them does something irreversible in production, opens a pull request against a payments service, or rewrites an IAM policy at 02:00, you’ll want to know exactly what it did, what it was running, and who authorised it. Most teams running agents today have nothing to pull from the wreckage.
The conversation we keep avoiding
We’re having a loud conversation about agent capability: which model codes best, how many tools it can chain. All worth discussing. A second conversation decides whether agents get anywhere near regulated systems, and almost nobody’s having it: accountability after the fact.
When an autonomous system does something you didn’t expect, can you reconstruct what happened from evidence you trust?
Aviation had this exact argument. For decades, pilots fought the cockpit voice recorder as a spy in the flight deck. The industry made it mandatory and crash-survivable anyway, and accident investigation was never the same. Cloud and DevSecOps is at the start of the identical reckoning. The plumbing already got standardised: Anthropic’s MCP for connecting agents to tools, and Google’s A2A for agents talking to each other, the latter donated to the Linux Foundation in December 2025. In February 2026, NIST’s Center for AI Standards and Innovation launched a formal AI Agent Standards Initiative built around security and identity. The recorder, the thing that tells you what the agent actually did, still has no standard at all.
The recorder runs on every flight, not just the doomed ones
A flight data recorder doesn’t wait for a problem. It runs from engine start to engine stop on every single flight, capturing hundreds of parameters whether the flight is routine or catastrophic. The recorder that mattered at Tenerife had been running all day, on flights nobody remembers, before it captured the eight seconds that did.
Agent instrumentation tends to work backwards. Teams log heavily around known failure modes and lightly everywhere else, then discover that the run which caused the incident is the one with the thinnest trail. The decision you most need to inspect happened in the part of the run you decided wasn’t worth recording.
The action: emit a complete provenance record on every run, success or failure, before you know which ones you’ll need. The cheap runs you record and never read are the price of having the one run you do need.
A recorder that melts in the fire is worthless
The reason flight recorders survive an 1,100°C fire for 30 minutes and an impact around 3,400 g is that the data is only useful if it outlasts the event it recorded. A recorder destroyed by the crash records nothing about the crash. That sounds obvious until you look at how agent logs are stored.
If your agent writes its trail to the same environment it operates in, a run that corrupts that environment can take the evidence with it. An agent with write access to a logging bucket can edit its own history. An agent that crashes the host loses whatever it hadn’t flushed. The trail has to live somewhere the agent’s worst run can’t reach.
The action: sign each record (a DSSE envelope over an in-toto statement does this with tooling your supply-chain pipeline already runs) and store it outside the agent’s blast radius. A signed record is tamper-evident: if anyone alters it after the fact, verification fails, which is exactly the property a black box gives an investigator.
The voice recorder and the data recorder tell different halves of the story
Aircraft carry two recorders for a reason. The cockpit voice recorder captures what the crew said and did. The flight data recorder captures what the aircraft was doing: airspeed, attitude, control inputs. You need both, because the same control input means different things depending on what the aircraft was actually doing at the time.
Agents have the same two halves. There’s what the agent did in a given run: which tools it called, which sources it read, what it changed. And there’s what the agent was made of when it did it: which model version, which skills, which MCP servers, which policy. A poisoned skill or a swapped model turns an innocent-looking action into an incident, and you can’t see that from the run alone.
The supply-chain version of this is already routine outside AI. When the xz compression library was backdoored in 2024, the compromised release looked legitimate enough to reach major Linux distributions before anyone noticed. Pinning components by digest is how you bound that kind of blast radius: a poisoned skill or a swapped MCP server has a different hash from the one you trusted, so a recorded digest tells you afterwards exactly which runs touched the bad version.
The action: pair every run record with an agent bill of materials (an AI-BOM) that pins each component by digest. When a skill or an MCP server is later found compromised, the digest lets you find every run that touched it instead of guessing.
Never let the pilot write his own black box
Here’s the detail from Tenerife that should worry anyone deploying autonomous systems. After the crash, KLM’s own review pushed back hard on the conclusion that its captain had taken off without clearance. The airline had every incentive to spread the blame across the fog, the radio interference, the crowded airport. The cockpit voice recorder didn’t have those incentives. It simply played back what was said, and what it played back made van Zanten’s decision undeniable.
This is the part most agent observability gets wrong, so I’ll say it plainly. Your observability stack is not provenance. Traces the agent emits about its own behaviour are a confession it gets to write. If the agent, or a component the agent controls, authors its own audit trail and nothing signs it independently, what you’ve got is the operator’s statement, and the operator is the thing under investigation.
The fix is the same one aviation reached: the recorder is independent of the operator. The record should be signed by the runtime or the workload identity that hosted the run, not asserted by the agent. Keyless signing tied to a workload identity makes the signature say “this environment ran this,” which is a claim an auditor can check.
The action: sign provenance with the runtime’s identity, not the agent’s self-report, and verify that signature before you trust a single field in the record.
The recorder remembers who had the controls
Crew resource management, the discipline aviation built after a run of accidents where junior officers failed to challenge senior captains, depends on knowing who had control and who said what. The recorder captures that. At Tenerife, it captured the KLM flight engineer questioning whether the Pan Am aircraft was clear of the runway, and the captain pressing on regardless. The chain of authority is in the recording.
Agents increasingly act with humans somewhere in the loop: a person approves a risky step, a reviewer signs off on a deploy. If that approval lives in a Slack thread and the action lives in a git log and nothing connects them, you can’t prove the approval covered the action that actually shipped.
The gap shows up the first time an auditor asks a plain question: show me that a human approved this specific change, not a similar one from the same afternoon. When the approval and the action sit in different systems with no shared identifier, the honest answer is that you’re inferring the link rather than proving it.
The action: bind each human approval to the exact action and the exact artefact it authorised, inside the same signed record. “Bola approved opening this PR at 09:12” belongs next to the PR’s own hash, not in a separate system you have to correlate by hand later.
Recorders were mandatory before the next plane took off
Aviation didn’t make recorders optional and hope operators would adopt them. It made them a condition of flying. No serviceable recorder, no commercial flight. The mandate came before the next inquiry rather than after it, because the whole point of a recorder is that you fit it before you need it.
Most agent deployments have this backwards. Provenance, if it exists at all, is something the team promises to add after the pilot ships, or after the first incident. By then the runs you most wanted to inspect have already happened, unrecorded.
Borrowing the levels model from software supply-chain security makes this enforceable. A run can be recorded but unsigned (fine for development), signed with a pinned bill of materials (fine for production), or signed and published to a transparency log with an identity-bound signature (the bar for regulated and cross-organisational work). Pick the level each environment requires and gate on it.
The action: make a provenance level a deployment gate. No record at the required level, no production. Treat it the way you’d treat a missing recorder on an aircraft about to carry passengers.
The best operators read the recorder when nothing went wrong
Aviation’s most useful move with recorders happened far from any crash site. Flight data monitoring, where airlines analyse recorder data from thousands of normal flights, catches the small drifts (an approach flown slightly too fast, a procedure quietly skipped) long before any of them line up into an accident. On the back of that, the recorder became a prevention tool, read most often on flights that landed without a story to tell.
Provenance has the same second life. If you only ever open an agent’s record after something breaks, you’re using a fraction of it. The same signed records, read in aggregate across normal runs, show you when a policy started getting bypassed, when an agent quietly began reaching for a tool it didn’t used to, when a model version changed under you.
The action: read provenance on a schedule, the way airlines read flight data from flights that landed safely. The drift you catch in a boring week is the incident you never have to write up in a bad one.
The only question that matters
Aviation learned, at a cost it has never forgotten, that an autonomous-enough cockpit needs a tamper-evident record the operator doesn’t get to write. It also learned that the moment of maximum danger is the one where everyone trusts the operator too much to question him, which is precisely the moment no one is watching.
Your most autonomous agent is the one you’re least likely to instrument, because it’s the one you trust. That’s the same instinct that put the airline’s best pilot at the controls with no one willing to challenge him. If you run agents in production right now and you couldn’t produce a signed, verifiable record of what your most powerful one did last Tuesday, you’re flying Tenerife’s approach and hoping the fog holds off.
So here’s the question worth sitting with: when your best agent does something irreversible, what plays back?
I’ve published a full v0.1 specification for agent run provenance, Open Agent Provenance, including the field set, conformance rules, and a worked example, at blog.ogunlana.net. If you’re putting agents anywhere near regulated systems, start there.