The Vendor You Can't Vet
In my line of work, we have a name for an entity that touches your data, writes your code, makes decisions on your behalf, and operates with almost no contractual accountability.
We call it an unmanaged third party.
And for the last decade, my entire career has been built on a simple premise: unmanaged third parties are how organizations get breached.
So when I read Matt Shumer’s viral article earlier last week, the one that racked up tens of millions of views and circulated through corporate leadership channels and industry group chats alike, I understood the reaction. The demonstration was undeniably impressive. But while the internet saw a productivity breakthrough, I looked at it through the specific lens of my profession, and I didn’t feel excitement. I felt something closer to professional alarm.
Not because Matt Shumer was wrong about what AI can do. He was largely right. But because buried in his breathless celebration of an AI that builds apps, tests its own work, and exercises something he called “judgment” was a description of the single most consequential vendor relationship most organizations will ever enter. And no one in the conversation seemed to notice that the vendor had never been vetted.
The Invisible Subcontractor
In construction, every general contractor understands a concept that keeps the entire industry honest: the subcontractor chain.
You hire a firm to build your office. That firm is bonded, insured, licensed. You’ve reviewed their track record. Your legal team has reviewed the contract. There are penalty clauses, liability frameworks, inspection schedules. You know who is doing the work, and you know who is accountable when something goes wrong.
But what happens when that firm quietly subcontracts the electrical work to someone you’ve never met? Someone with no license you can verify, no insurance you can audit, no track record you can review? Someone whose methods are, by their own admission, not entirely explainable even to the people who hired them?
In construction, that’s a liability nightmare. In cybersecurity, that’s Tuesday.
Here is what Shumer described in his article, stripped of the wonder and rewritten in the language of vendor risk management:
He engaged a third-party system to autonomously write tens of thousands of lines of code. That system then self-tested its own output, made independent design decisions about user experience, iterated on its own work without human oversight, and delivered a finished product that Shumer accepted without modification. He described this process happening while he was away from his desk for four hours.
He was not describing a productivity breakthrough. He was describing a fully autonomous, unsupervised vendor engagement with zero human-in-the-loop review, no quality gate, no code audit, and no accountability framework.
And he was celebrating it.
In the world I come from, what Shumer described is the largest unvetted vendor onboarding in corporate history. And it is happening everywhere, all at once, with almost no one from risk management in the room.
The Trust Problem
I’ve written before about the transformation of organizational security from the castle model to the city model. The idea that you’re no longer the Lord of the Castle, surrounded by walls you control, but the Mayor of an Open City, surrounded by vendors, partners, and service providers whose security posture directly affects yours.
That metaphor still holds. But AI introduces something the city model didn’t fully anticipate.
In the traditional vendor ecosystem, you at least know who your vendors are. You have contracts. You have SLAs. You can send them a security questionnaire (whether they fill it out honestly is another conversation). You can tier them by risk. You can monitor them. You can, if things go badly enough, terminate the relationship.
AI as it exists today breaks almost every one of those governance mechanisms.
Consider the fundamentals of any serious vendor risk program. You need to know what the vendor does with your data. But the training data pipelines behind large language models are opaque even to the companies that build them. You need to understand the vendor’s security controls. But the internal architecture of these models is, by design, a black box. You need the ability to audit. But you cannot audit a system whose decision-making process is not fully interpretable, even to its creators. You need contractual accountability. But the terms of service for most AI tools disclaim liability for output accuracy, security, and fitness for purpose in language that would make any procurement lawyer’s eye twitch.
And then there is the line from Shumer’s article that I keep coming back to. The line that OpenAI included in the technical documentation for their latest model: that GPT-5.3-Codex was “the first model that was instrumental in creating itself.” That the Codex team used early versions of the model to debug its own training, manage its own deployment, and diagnose test results and evaluations.
Read that from a supply chain perspective and it sounds like this: the vendor you are trusting to write your code was itself built with the assistance of a previous version of itself, using processes that are not fully documented, not fully auditable, and not fully understood.
In vendor risk management, we have a phrase for this kind of arrangement.
We call it “hope.” And, as I said in my previous article, hope is a strategy for the unprepared.
The Judgment Question
Shumer’s article made a claim that deserves careful scrutiny, not because it’s necessarily false, but because the implications are enormous if it’s even partially true.
He said the latest AI models had something that felt, for the first time, like judgment. Like taste. An intuitive sense of knowing what the right call was, not just the technically correct one.
I want to take that seriously for a moment. Not to dismiss it and not to accept it uncritically. But to ask a question that I don’t think enough people are asking: If an AI system genuinely exercises judgment, who is accountable for the judgments it makes?
In every other vendor relationship, the answer is clear. If your outsourced development team makes an architectural decision that introduces a vulnerability, there is a contractual chain of responsibility. If your MSSP misses an alert that leads to a breach, there are SLAs, liability clauses, and insurance policies that define who answers for what.
But when AI exercises “judgment” about your code, your user experience, your data handling, your business logic… the accountability chain dissolves into fog.
The AI provider’s terms of service almost certainly disclaim responsibility for the quality of the output. Your internal team accepted the output without modification (Shumer himself said corrections were rarely needed). The decision was made by a system that cannot be deposed, cannot be cross-examined, and cannot explain its own reasoning in terms that would satisfy a regulatory inquiry.
This is what organizations are doing right now, at a scale that grows every week, while the risk management function looks the other way because the productivity gains are too compelling to question.
You can outsource the labor. You cannot outsource the liability.
That line appears in my books because it captures the fundamental truth of third-party risk. It was written about human vendors, about outsourced IT teams and cloud service providers and managed security operations. But it has never been more relevant than it is right now, applied to AI.
The organizations deploying AI at the level Shumer celebrates are outsourcing cognitive labor at a scale and speed that has no historical precedent. And many of them have not paused to ask the question that every vendor risk professional is trained to ask first: If this goes wrong, who is responsible?
The Self-Assembling Bridge
Let me offer a metaphor that I think captures the unique challenge AI poses to traditional governance frameworks.
Imagine you are a city engineer. One morning you arrive at your office to find that someone has built a bridge overnight. It is beautiful. Structurally impressive. It carries more traffic than any bridge your team has designed. Cars flow across it smoothly. The pedestrian walkways are elegant. The load distribution is remarkable.
There is just one problem. The bridge designed its own load-bearing structure using methods your engineering team cannot fully explain. They can test it. They can drive trucks across it. They can measure the stress on the cables. But they cannot audit the blueprint, because the blueprint was generated by a process that is not fully transparent, even to the entity that created it.
Now imagine someone tells you to build your city’s entire infrastructure on bridges like this. And that it’s already happening in cities around the world. And that slowing down means falling behind.
That is the position that every CISO, every CRO, every board member with fiduciary responsibility is in right now. The bridges work. They work impressively. And the pressure to build more of them is intense. But the fundamental question of auditability, the question that underpins every regulatory framework, every compliance standard, every governance structure we have built over the last twenty years, remains unanswered.
I am not arguing that we should refuse to cross the bridge. I am arguing that we should not pretend the engineering review happened when it didn’t.
What the Viral Conversation Missed
Shumer’s article generated tens of millions of views and thousands of responses. The discourse split predictably. One camp said this changes everything. The other camp said this is overhyped. Both camps were arguing about the wrong thing.
The question is not whether AI is powerful. It is. I use these tools. I see what they can do. I am not standing on the shore shaking my fist at the tide.
The question is whether our governance frameworks have kept pace with our adoption speed. And the honest answer, the answer that no one in the viral conversation wanted to engage with, is: they haven’t. Not even close.
Think about how long it took organizations to build mature vendor risk programs around cloud computing. The first serious enterprise cloud migrations started in the early 2010s. It took the better part of a decade for most enterprises to develop governance frameworks that could meaningfully assess cloud provider risk. And cloud computing, for all its complexity, is fundamentally auditable. You can inspect the architecture. You can review the configurations. You can conduct penetration tests. You can contractually define responsibilities.
AI adoption is moving ten times faster than cloud adoption, into a technology that is orders of magnitude harder to audit. And the governance gap is not shrinking. It is widening with every deployment.
Here is what I would have wanted to see in Shumer’s article, or in the thousands of responses it generated. Not less excitement about what AI can do, but more seriousness about what AI requires.
** Where is the vendor risk assessment for the AI writing your production code? ** Where is the data classification review for the information being fed into these models? ** Where is the supply chain map that traces the training data, the fine-tuning process, the model architecture, and the deployment pipeline? ** Where is the incident response plan for when the AI’s “judgment” produces a decision that creates legal exposure? ** Where is the third-party audit? ** Where is the board-level oversight?
For most organizations, the answer to every one of those questions is the same: nowhere. It doesn’t exist. We skipped it because the technology was too exciting and the competitive pressure was too intense.
We are building on the bridge and calling the engineering review a problem for next quarter.
The Governance Gap Gets Wider
I called the final volume of my Vendor Risk Leadership Series The Governance Gap because I was seeing a pattern across every industry I consulted with: the distance between what organizations were doing with third-party technology and what their governance structures could actually oversee was growing faster than anyone wanted to admit.
That gap just got exponentially wider.
AI is not just another vendor in the ecosystem. It is a vendor that is being embedded into every function, every workflow, every decision chain in the organization. It writes code that touches customers. It analyzes data that informs strategy. It drafts communications that carry legal weight. It makes recommendations that shape business outcomes. In Shumer’s telling, it exercises judgment.
And it does all of this with governance oversight that would be unacceptable for a $50,000 software contract, let alone a technology that is reshaping the operational core of the enterprise.
The board members reading Shumer’s article (and many did; Fortune syndicated it) should not have been asking “how do we go faster?” They should have been asking “do we have a framework for governing this?” Because if the answer is no, and for most boards the answer is no, then the fiduciary exposure is real and growing.
I keep returning to a truth that has guided my work for years: you can outsource the labor, but you cannot outsource the liability. Every vendor processes your data, hosts your applications, touches your customers. Every one of them is a door you may not know is open.
AI is not just another door. It is a door that is building other doors. And it is doing it while most organizations are still debating whether they need a policy.
So What Do We Actually Do?
I am not here to tell you to stop using AI. That ship has sailed, and truthfully, it should have sailed. The capabilities are real. The productivity gains are real. Ignoring these tools would be its own kind of failure.
But the way we are adopting them is reckless, and it doesn’t have to be.
If you lead a security function, start by treating AI systems the same way you treat any critical third-party provider. Tier them. Assess them. Build controls around them. But be realistic about the limitations of your current toolkit. We cannot just send a security questionnaire to an LLM. We need new controls, like mandatory human code review, sandboxed execution, and output-based testing, that replace the traditional vetted trust.
If your organization would never deploy an outsourced development team without a security review, then it should not deploy an AI coding assistant without one either. The fact that the vendor is a model rather than a company does not change the risk. It changes the attack surface math.
If you sit on a board or serve in a governance role, ask your CISO a simple question: “Do we have a vendor risk framework for AI?” If the answer is a long pause followed by a restatement of the productivity benefits, you have your answer. The governance gap is real, and it is your exposure.
If you lead a team that is adopting AI into its workflows, insist on a human-in-the-loop review for anything that touches production systems, customer data, or business-critical decisions. Shumer celebrated walking away for four hours and returning to a finished product. I understand the appeal. But in a mature security posture, the absence of human review is not a feature. It is a control failure.
And if you are reading Shumer’s article, or the dozens of pieces like it that will follow over the coming months, read them with a question that the authors rarely ask: Who is accountable when the magic doesn’t work?
Because it will happen. Not because AI is bad, but because all systems fail. All vendors have incidents. All technology introduces risk. That is not pessimism. It is the entire reason the disciplines of vendor risk management, supply chain security, and governance exist.
The Question Worth Sitting With
I started this piece with a claim: that Matt Shumer’s article describes the largest unvetted vendor onboarding in corporate history. I want to end with something more uncomfortable.
It is not just Shumer. It is all of us.
Every organization deploying AI at enterprise scale without a governance framework is doing exactly what Shumer described. Trusting without verifying. Accepting output without auditing process. Celebrating speed without questioning accountability. Treating a profoundly powerful and profoundly opaque technology as though it were a tool rather than a partner, a calculator rather than a decision-maker.
We know better. Every lesson of the last two decades of cybersecurity tells us what happens when trust outpaces verification. Every breach, every supply chain attack, every third-party incident that made headlines did so because someone, somewhere, decided that the benefits were too obvious to question.
The benefits of AI are obvious. I will not pretend otherwise.
But obvious benefits have always been the most effective cover for invisible risks. And the risk here is not that AI will fail spectacularly. The risk is that it will succeed just well enough, for just long enough, that we never build the governance frameworks we need until after the incident that proves we should have.
So here is my question for the leaders reading this. Not the question about whether AI is transformative, because it is. Not the question about whether you should adopt it, because you should. But this one:
If the AI you deployed today were a human vendor, and it did exactly what it does, the same access, the same autonomy, the same opacity about its methods, would your vendor risk program approve it?
If the answer is no, then you have work to do. And the work is not to slow down. The work is to govern what you’ve already accelerated.
The castle walls are gone. The city is open. And the newest resident, the one who builds faster than anyone, never sleeps, and answers to no one’s procurement office, just moved in.
The question is not whether to welcome it.
The question is whether anyone checked its references.
Julio Bandeira de Melo is the author of The Vendor Risk Leadership Series, including The Invisible Perimeter and The Digital Backdoor. He is a Senior Manager in Security Consulting with over twenty years of experience building security programs for global enterprises. He writes about supply chain cybersecurity, leadership, and the invisible work of governance. Subscribe to his newsletter and publication for clarity in chaos.