For most of my career, doing something serious with a computer meant paying for it first, in time. Learning to configure DNS costs a weekend and one broken domain. Getting SPF, DKIM, and DMARC right costs a week of mail that bounces for reasons nobody explains. That cost acted as a gate, and if you had cleared it, you had already learned something about what goes wrong.
That cost is mostly gone now. An LLM assistant will walk someone with no operations background through renting a server, pointing a domain at it, installing a service, opening the firewall port, and putting the result on the internet, in an afternoon. That is mostly good news, but it is also a source of risk. The knowledge the gate filtered turns out to have been of two kinds, and only one was technical.
The gate was never a security control. It was friction, and the friction did two jobs: keeping unqualified people out, and, by accident, keeping a large amount of half-configured software off the public internet.
The expertise was rarely about syntax. Most of the skill in this trade is a mental model of failure modes, which shows up as a set of reflex questions. What happens if this runs twice? What does this delete if the variable is empty? Row-level security, DNS propagation, rsync --delete, DELETE without a WHERE: none of these are interesting, and none are learned for fun. They are learned by causing the incident.
So the gate filtered some risk, but not the kind you would choose to filter. It filtered unawareness rather than incompetence, a self-taught person with a shell being no safer than a trained one, and it never worked reliably anyway. Recently I watched a marketer ask Claude to build a market analysis tool. Claude complied and produced a working prototype. The marketer then asked for an overseas team to be given access to it. Claude complied again, raising a mild objection about permissions before granting access without further verification. The marketer did not realize that the objection was the important part, having never spent a career worrying about access control.
After spending half a decade working on the codebase at my previous employer, I was acutely aware of the non-standard ways guest accounts were handled. Not just the technical implementation, but also the business motivations behind them. In reporting, these subtleties mattered, but my agent was unaware of any of this context and would produce incorrect reports. Fortunately I reviewed these reports and caught the discrepancies, but if someone from accounting or marketing had asked their agent to produce a report, they would probably not have caught it.
The person who could build the tool was usually also the person who had been in the room for years. They knew what mattered the way everybody does: listening in meetings, arguing at lunch, noticing which requests keep coming back, hearing the enthusiasm in the room not match the announcement. Tacit knowledge is the term for this, and its defining property is that it is not written down anywhere, partly because the people holding it cannot fully articulate it.
An agent has no access to any of that. It knows what is in its context window and nothing else. It does not know that the same request was attempted in 2019 and died over a customer contract, that the dashboard being asked for is the pet project of somebody who leaves in June, or that a process is manual because the integration partner will not sign an API agreement. A longer prompt does not fix this, because the information does not exist in written form to be included.
The result is not that the software gets worse. It is that the software gets better and answers the wrong question. Capability and judgement about what is worth building used to arrive together, and only the first has been automated. Requests still have to come from somewhere, and now they come from whoever can type them: more people than before, several of whom are closer to their own problem than to the company’s. A prototype built that way is not exactly wrong. It is a correct answer to an incomplete question, with a URL and a database attached.
Reviewing what the machine produced is the correct first answer. Georgia Tech’s advice is to review AI output the way you would review a junior developer’s pull request, with extra attention to input handling and authentication. Nobody should ship without that, but it assumes a review process that works, which I am not convinced it does here.
Agents do not fail the way people fail, and our review instincts are calibrated on the way people fail. When you review a colleague’s code you get scaffolding for free: how senior they are, which parts they rush, and the option of asking them. JetBrains and Lund University call the generated alternative heterogeneous output presented with homogeneous confidence, so the trivial authentication change and the risky migration read exactly the same, and the diff tells you nothing about where the risk sits. Their conclusion is that reviewing agent-written changes is not a diffing problem but a trust-calibration problem.
The failure modes differ as well. Human bugs cluster where humans are sloppy, in boundary conditions and forgotten error paths, which is exactly where reviewers look. A model’s weak spots do not announce themselves. It writes uniformly idiomatic code whether it is right or wrong, hallucinates packages that never existed, and hides its mistakes in the context it assumed rather than in the code it wrote: generated Terraform can be flawless and pointed at the wrong account. Those failures are not visible in the change, because the error is not in the code.
The empirical version is not encouraging. A study with 100 participants gave developers a choice between AI suggestions that varied in security and functionality, and found no difference in selection rates between the most and the least secure option. Few did anything resembling a security review; many went on visible cues, such as an edge-case block they could see, or on style. Those who edited the suggestions more heavily ended up with more secure code, so they could fix the problem once they saw it. Seeing it was the hard part.
I feel the industry is still in the early stages of understanding how to handle these challenges. We’ll need to develop new practices, tools, and mindsets to ensure that the benefits of AI agents are realized without compromising safety, security, and correctness.
I feel we should focus our reviews on LLM input instead of output. The prompts, context, and instructions given to the model are as critical as the code it generates. Modern models don’t really make major mistakes, unless instructed incorrectly, either explicitly or through omission. Ensuring that the input aligns with security, policy requirements, and company goals can prevent many issues before they arise. A meta-agent might review the input while it is being typed, and flag problematic inputs before they are executed, but it should not get in the way of agent operation, as at that point the user will be motivated to circumvent it.
At the same time, we will need a new kind of access control, specifically to allow agents to act on behalf of a user in a traceable manner. Agents should not work with user credentials, but with derived delegations. These delegations need to be visible to the user, auditable by a security team, and revocable by both. LLMs should be designed to request these delegations, ensuring that any action taken on behalf of a user is properly logged, scoped to the necessary permissions, and can be reviewed.
The gate is not coming back, and reinstating it is not a serious option: it used obscurity as a form of control rather than addressing the underlying issues.
The alternative to the gate is early, and it is worth being honest about what that means. The proposals above are not practices to adopt, they are things that mostly do not exist yet: review that lands on what you feed the model rather than on what it produces, and access control built around delegations an agent can request, and that a human can audit and revoke afterwards. Both point the same way, which is that authority should be granted per action and be visible, rather than carried around in whatever credentials happen to be loaded. In the meantime the fallback is what it has always been, a person who understands the change looking at it before it lands, with the caveat the previous section raises.
Note also what is missing from that list. Both proposals are about safety, and neither is about relevance. Nothing in them helps an agent work out that the dashboard being requested is the pet project of somebody who leaves in June. Input review catches an instruction that violates policy; it does not catch an instruction that is pointless, because that is not a control problem, it is a judgement problem, and the judgement comes from having been in the room.
The bottleneck moved from “can you build this” to “can you judge this”, and the judging is twofold: whether it is safe, and whether it is the right thing to have built. For the second question there is no prompt that substitutes for having been there. The tools opened up; the consequences did not get any smaller.
A practical guide to replacing Cloudflare’s services with EU-based alternatives, covering CDN, DDoS protection, edge hosting, and why you might want to make the switch.
Read More
A practical look at building a side project entirely through AI-assisted “vibe coding”, with observations on where it excels and where it falls short.
Read More
A reflection on how AI is transforming software development, drawing parallels to the Industrial and Digital Revolutions that displaced previous professions.
Read More