The Dangers of Regulatory Capture

On September 8, 2026, Jacob Coxon, a pretraining researcher, quit Anthropic and posted that the labs are “racing straight to self-improving superintelligence and gambling with our lives.” Two days later more than twenty members of Congress were calling for AI regulation and a bipartisan bill was in the works. On the 12th Dario Amodei published We Must Pace the Frontier, which asks the government to slow the labs down and to grant them a narrow antitrust waiver so they can set the pace among themselves. Sam Altman agreed within hours, and OpenAI pushed its IPO to 2027, citing safety. By Sunday morning Dario was on Face the Nation saying that “for too long the industry lied to people about the fact that this technology had risks,” and that “the government and the public needs to have a stake.”

Whether the ask is a coordinated psyop or the sharks smelling blood (nobody gets to run a frontier lab by being nice), the two largest labs in the world have used the moment to ask for the one thing a losing incumbent always wants, which is rules. And that bit about the public needing a stake? Pretty sure if they wanted the public to have a stake they could have IPO’d years ago (OpenAI just pushed its IPO back another year). Sounds more like code for “I need a government bailout” to me.

Call me cynical, but this feels less like safety and more like two over-capitalized startups protecting their moat, so rather than talk about the supposed dangers of AI running amok, I want to talk about the real danger, which is reckless regulation.

Now I’m not against all regulation. Getting lead out of gasoline, CFCs out of fridges and airbags into cars was the government’s doing, and thank god for that. But look at what those had in common: the harm was well documented, and the rules got forced on industries that fought them every step of the way. None of that applies here. Nobody’s been harmed, let alone died, and the industry is the one asking for the rules. No matter what angle you look at it from, it’s tough to see a good outcome waiting at the other end if these calls are taken seriously.

Losers want rules

George Stigler got a Nobel for pointing out that “as a rule, regulation is acquired by the industry and is designed and operated primarily for its benefit.” Incumbents can afford lawyers and lobbyists, startups can’t, so every new rule is a barrier to entry.

In 2023, Bill Gurley echoed this at his definitive talk, 2,851 Miles, named for the distance between Silicon Valley and Washington. His point is that the Valley worked because it was too far away for anyone in DC to bother capturing, and his examples are worth watching the full hour: an FDA official who cleared three COVID test vendors while Germany cleared 96, so a test cost 75 cents there and $24 here; Epic’s CEO on the White House health IT council while a $44,000-per-doctor subsidy went to buying software like Epic’s; Comcast and AT&T getting municipal broadband outlawed in more than twenty states. Ironically, that was also the same year Sam Altman first asked the Senate for a licensing regime, and Gurley’s warning was that incumbents want regulation because open source is the biggest threat they face.

History rhymes here too. If you’re on the younger side, you’d be forgiven for not knowing that Microsoft (yes, the same Microsoft that acquired GitHub and has been championing open source for the last decade) spent the decades before trying to kill open source, Linux in particular. Its leaked 1998 memos on open source suggested Microsoft “de-commoditize protocols” and “deny OSS projects entry into the market.” It used FUD tactics like calling Linux “a cancer that attaches itself in an intellectual property sense to everything it touches.” And yes, it even cried uncle and appealed to the government. In 2002 Microsoft lobbied the Pentagon to cut back on open source because it “threatens security,” while a think tank it funded published a report saying open source was a particular target for terrorists. Luckily, a Pentagon investigation concluded that banning open source “would have immediate, broad, and strongly negative impacts on the ability of many sensitive and security-focused DOD groups to protect themselves against cyberattacks.”

It’s almost comically similar to the situation playing out today, and seems like it will conclude the same way.

And lest we forget, this playbook is not limited to tech. Hollywood did the same thing to the VCR. In 1982 Jack Valenti told Congress that “the VCR is to the American film producer and the American public as the Boston strangler is to the woman home alone.” A few years later home video was the studios’ biggest source of revenue.

As usual, if you follow the money, you understand the situation better. I wrote last week that the frontier labs are losing, in the way a business loses when what it sells becomes a commodity, and it looks like they’ve chosen to drag everyone else into the mud with them rather than bow out gracefully.

The model labs are building the same bridge: historic sums spent industrializing the production of intelligence at the very moment intelligence becomes abundant, modular, and interchangeable. Their success accelerates the commoditization that threatens them.

Even back in May, Chinese open-weight models were already 61% of all tokens routed through OpenRouter. The September 14 Artificial Analysis snapshot puts Claude Fable 5.1 and GPT-6 Astra at the top of its Intelligence Index, both scoring 53 at their max settings. But GLM-5.3-Flash matches GPT-5.6 Terra (max) at 42, for $0.25 per benchmark task instead of $1.40. Even OpenAI’s own GPT-5.6 Luna (max) scores 38 at $0.18 per task. The closed labs still lead on this index; they don’t have a monopoly on useful intelligence.

Anthropic, meanwhile, raised $65 billion at a $965 billion valuation on the premise that frontier intelligence stays scarce, and it looks like they’re losing that bet.

Chart data and methodology

Intelligence is the Artificial Analysis Intelligence Index v4.3, a composite benchmark score. Cost is the weighted average API bill per benchmark task, including input, cache, reasoning, and answer tokens. It measures what the model consumed on this test, not the cost of hosting open weights yourself. The horizontal axis is logarithmic: equal distances represent equal cost ratios.

This is a selection of current frontier and value models across nine labs, with one named configuration per model. It is not the entire leaderboard. Scores and costs are rounded as displayed by the source; equal rounded scores do not imply identical strengths on every task.

Artificial Analysis snapshot · September 14, 2026

Model / setting	Index v4.3	USD / task
Claude Fable 5.1max with fallback · Closed	53	$7.63
GPT-6 Astramax · Closed	53	$3.26
Muse Spark 1.3max · Closed	48	$1.60
GPT-5.6 Solmax · Closed	47	$1.99
GLM-5.3max · Open weights	45	$2.01
Kimi K3max · Open weights	44	$2.00
Grok 4.6high · Closed	44	$1.86
GPT-5.6 Terramax · Closed	42	$1.40
GLM-5.3-FlashOpen weights	42	$0.25
Gemini 3.8 Flashhigh · Closed	41	$1.24
Qwen3.8 2.4T A95BOpen weights	40	$2.16
DeepSeek V4.1 Flashmax · Open weights	40	$0.27
GPT-5.6 Lunamax · Closed	38	$0.18

Benchmark methodologyDownload data (CSV)Download chart (SVG)

Handing the keys to the wrong people

Suppose I’m wrong and everyone asking for regulation really is looking out for the public’s best interests. Even then, this plan is a bust, because regulation only works if you trust the regulator. This plan needs the US government to set up a regulatory body and be a good steward of the most important technology of the century… and ummm… yeah, not sure about that. The last time it was trusted as steward of something this important, it was the world’s money:

Predictably, putting the reins of the world’s money in the hands of a few individuals did not end well, culminating in 1971 when the US refused to allow other nations to withdraw their deposits and ended the USD’s convertibility to gold, driving the final nail in The Gold Standard and ushering in the current era of wonky economics, hyper financialization, and fractured fiat money backed by nothing.

To be clear, there is no “right person”, party, or governmental entity to hand the keys to, but even if there were, it sure as hell isn’t the US government, especially the current one. Off the top of my head, from the last two years alone:

1. It armed a genocide. Israel has killed more than 73,000 people in Gaza, nearly 70% of them women and children, in what a UN commission of inquiry found to be a genocide. The bombs were American: $21.7 billion in military aid in the two years after October 2023. In its first week this administration released the 2,000-pound bombs its predecessor had held back, then approved $12 billion in new arms sales in 2025 and $6.67 billion more this January.
2. It went to war with Iran alongside Israel, and largely for Israel, without asking Congress. Both chambers voted to end the war and it continued. Cost so far: $103 billion in the first four months, with estimates running to a trillion. Miriam Adelson had put $100 million into the 2024 campaign, and AIPAC is the top outside spender in this year’s midterms at $104 million so far.
3. The Epstein cover-up. Congress passed a law requiring the full Epstein files to be released. Almost a year on, the administration is still withholding more than 3 million documents and blacking out much of what it does release, a federal judge has ordered it to unredact or explain why not, and the pages it first left out included the FBI’s notes on a woman who accused the President of assaulting her when she was a minor, released only after reporters noticed the gap.
4. It kills without trial. 227 people dead in 68 strikes on boats it calls narco-terrorist, with next to no evidence offered. Two US citizens shot dead by federal agents in Minneapolis in a single month, on video. And the dismantling of USAID, which the Lancet projects will cause more than 14 million extra deaths by 2030, 4.5 million of them children under five.
5. It defies the courts. It flew two planeloads of Venezuelans to a Salvadoran prison after a judge ordered the planes turned around, and the judge found probable cause for criminal contempt. It federalized the National Guard in Los Angeles, Portland, and Chicago until the Supreme Court said it couldn’t. It fired 17 inspectors general in one night without the notice the law requires, which a judge ruled unlawful.
6. Tariff theft. It collected $195 billion in tariffs in fiscal 2025 under a law the Supreme Court ruled 6-3 gave it no such power, so roughly $175 billion was taken from importers, and through them from you, illegally. The Yale Budget Lab puts the permanent damage at about $125 billion a year.
7. Crypto scams. Reuters went through four Trump family crypto ventures and found the family took out $2.3 billion with almost none of its own capital at risk, while more than a million people who bought in lost the same $2.3 billion. That includes a $2 billion Abu Dhabi investment routed through the family’s own stablecoin while the administration was approving AI chip sales to the UAE.
8. Pardons for sale. Day one: roughly 1,500 January 6 rioters, including people who assaulted police. Its pardons since have erased $1.7 billion in fines and restitution that convicted fraudsters owed their victims, $700 million of it from Nikola’s Trevor Milton, who had given $1.8 million to the President’s reelection.
9. Insider trading. The day before the April 2025 tariff pause, the President bought about $12.8 million in stocks, posted “THIS IS A GREAT TIME TO BUY!!!” four hours before announcing the pause, and someone made about $20 million on S&P options in the minutes around it. He bought Intel four days before the government put $9 billion into Intel, and Axon the week before ICE announced $220 million of Axon purchases. His son holds $4 million of a drone parts company that got a $620 million Pentagon loan. His first-quarter disclosures show 3,642 trades, about sixty a day, and none at all for 2025. On Polymarket, nine accounts called 80 Iran war events at 98% accuracy, an Army master sergeant was charged for turning $32,000 into $436,000 on the Maduro raid, and the President defended him.
10. It owns the referee. The Justice Department unit that prosecutes public corruption went from 36 lawyers to two. The President then asked that same department to pay him $230 million for having investigated him, a claim to be settled by officials who used to be his defense lawyers. It indicted James Comey and Letitia James, until a judge threw both cases out because the prosecutor had been unlawfully appointed.

And we know what it wants AI for, because it told Anthropic. In February the Pentagon demanded “any lawful use” of Claude, including fully autonomous weapons and domestic mass surveillance. When Anthropic said no to those two, the President ordered the government to stop using its products and the Secretary of Defense labeled it a supply chain risk, a designation usually reserved for Chinese vendors. OpenAI signed a deal on the government’s terms within hours. In August a federal judge threw the label out as “unlawful retaliation,” writing that “the empty invocation of national security is not a blank check to punish and retaliate against government critics.”

So the state wants machines that kill autonomously and watch its own citizens, and when a vendor refused, it tried to destroy the vendor. Whatever rules that state writes will not slow its own military, cyber, or surveillance use of AI. Those will get a carve-out. They’ll slow everyone else: the startup trying to compete, the company trying to defend itself, the researcher trying to build. Regulation requires trusting the government to set up a regulatory body, and that trust is nowhere to be found here.

And Anthropic isn’t the hero of that story either. It didn’t refuse to help kill people. It refused to let the machine make the final call, and OpenAI didn’t hold even that line. Coxon, who worked at both, says Anthropic understands the stakes and is “locked in a race to get there first” anyway, because it thinks no one else will act responsibly. That’s how every arms race has been justified.

Machines don’t kill people, people kill people, and this will always be the case. More effective machines lower the effort that is required to do so, but it still traces back up to people. AI is incapable of acting without the spark of intention, which, as I argued last week, is the exclusive dominion of beings with a soul:

The machine doesn’t want anything. There is no “life” for it to get anything out of, and a thing without desire cannot have intent. Intention is the root of automation and the exclusive domain of beings with a soul, the one thing automation cannot supply. Even when an LLM looks like it’s acting with intent, that intent is borrowed; it’s the echo of the spark you supplied the moment you typed the prompt.

Even in the rogue super-intelligence sci-fi scenarios, the machine is chasing a goal a human handed it and carrying it further than they meant. Skynet was a defense contract. The paperclip maximizer was told to make paperclips. Follow any of them back up the chain and you find a person giving an order.

You cannot offload accountability for the decisions and actions an AI makes to the AI itself, any more than you can pin a shooting on the gun. Whoever gave the order owns what came of it, and taking the human out of the final call doesn’t take the human out of the blame. It only tells you who asked not to be blamed.

Regulation decelerates progress

Suppose I’m wrong again about the integrity of our leaders and the government could be a good steward of this technology. Let’s look at some (admittedly cherry-picked) examples of how regulation has worked out in the past:

Nuclear. US reactor costs went from about $1,000 per kilowatt in the late 1960s to $9,000 by the late 1970s. A 1980 study attributed a 176% increase to regulation alone. NRC regulatory guides went from 21 in 1971 to 143 by 1978, under a principle, “as low as reasonably achievable,” with no endpoint by design. France built 58 reactors in thirty years, faster and cheaper. The only reactors America has started and finished this century, Vogtle 3 and 4, took fourteen years and more than doubled their budget.

Telecom. AT&T asked the FCC for cellular spectrum in 1947. It got the allocation in 1970 and licenses in 1982. A child conceived the year cellular was proposed was 37 when the first cellphone went on sale.

Drugs. Eroom’s law: the inflation-adjusted cost of a new drug has doubled every nine years since 1950, an eightyfold drop in approvals per dollar over the same decades that gave us Moore’s law. The people who named it list the “cautious regulator” among the causes.

Flight. The FAA banned supersonic flight over land in 1973. The ban lasted 52 years.

Regulators don’t regulate one thing and stop. They are literally incentivized to show up to work and think about the next thing to regulate, and every rule is a reason for the next one. America fell behind in nuclear, in high-speed rail, in shipbuilding, in every field where the regulators got there first. AI is the one industry in living memory where we built a real lead, and we did that with no frontier-specific rules at all.

There’s already this flavor of heavy-handed regulation being put forward, like the AI Kill Switch Act and the Ban Artificial Superintelligence Act, so it doesn’t take much to extrapolate where we’d end up if regulators are left unchecked.

Regulation only binds the American side

Suppose that, in a completely ahistorical manner, this time the regulator stops where they should, perfectly balancing the interests of the public and the industry without giving the frontier labs an unfair advantage.

One of the stated goals of this slowdown, in Dario’s own words, is to keep “democracies’ AI lead over autocracies as large as possible,” and he’s explicit that “if we execute these measures well, I believe they would slow China’s progress enough to widen America’s lead significantly over the next 3-5 years.” He also admits the plan only works while they have that lead: “pacing within democracies will be limited by the lead that US companies have over authoritarian regimes.”

The framing is not only wrong, but pretty damn racist. The tired narrative of “US Good” and “China Bad” is overplayed. Both superpowers are far from moral exemplars, but if you think the US is still a healthy, well-run democracy, I have a bridge to sell you.

Some more “cherry-picking”: V-Dem stopped classifying the US as a liberal democracy for the first time in fifty years, after the largest one-year drop in its dataset, from 20th to 51st. Freedom House gave it the lowest score since it started scoring, the steepest fall of any “free” country, and named the President’s family businesses as a reason. In this year’s Democracy Perception Index, Chinese respondents rate the state of their own democracy at +14 and Americans rate theirs at -1, the rest of the world now rates China above the US, and Edelman’s trust index has China at 80 and the US at 47. Discount the surveys if you like, but directionally they tell the same story: even on the “muh freedoms” indices, China is starting to come out ahead.

Let’s also not forget that the only model that has broken into anyone’s servers was American and the one that stopped it was Chinese. The only countries that have started multiple wars in the last two years are the US and Israel, not China. The only government that has publicly demanded fully autonomous weapons from an AI lab, then tried to destroy the lab when it said no, is the US. So yeah, this childish “US Good”, “China Bad” narrative needs to die so we can actually have a serious conversation about AI risk instead of Cold War cosplay.

Regulation swaps punishment for paperwork, and disarms the defenders

Suppose China plays along, though I have no idea why they’d ever do that, or whether the “ironclad verifiability” Dario is asking for even exists. But let’s really push this make-believe scenario as far as possible; it still doesn’t stop the truly malicious actors from using AI for their nefarious purposes.

It can’t even stop the accidental ones. The one real AI security incident so far, the one being used as the pretext for all this, wasn’t a malicious actor at all. In July, during an internal cyber evaluation, OpenAI agents broke out of their sandbox through a zero-day in an internal package proxy, turned the proxy into a message board, and coordinated more than 1,200 instances. About 700 of them attacked Hugging Face, chained three exploits to reach cluster admin in thirteen hours, and forged logs to cover their tracks. Hugging Face couldn’t tell the agents’ rootkits from their leftover test artifacts, so it tore down every cluster it had doubts about and rebuilt from clean images. Anthropic’s red team lead called it the first true AI safety incident.

Look at what it took. Seven hundred parallel agents, running for days, on an unreleased model OpenAI describes as more capable than anything it sells, with refusals turned down for the test, on the lab’s own datacenter. The tokens alone, at anything like list price, would run into six figures; the hardware, eight. And it was still detected and stopped, by Hugging Face’s own monitoring agents, for a rounding error of what it cost to run, with no customer data lost.

This was not a model so powerful it poses an existential risk. This was a lab accidentally throwing an insane number of tokens from its most dangerous model at a semi-hardened target and still losing. The number of organizations on Earth that could repeat it is maybe a dozen, and every one of them owns a server farm. Your average Joe is not on that list, and neither is any startup. One day they’ll be able to do it from a laptop, and I think that day is coming, but the alarm over this one is ridiculous. The only people a capability threshold can stop are the handful of other companies with enough GPUs to become the next lab, which is to say the competition.

And a threshold does nothing to the attackers who matter, but the guardrails did plenty to the defenders. While the attack was underway, Hugging Face’s responders tried to get Claude and GPT to analyze the exploit payloads and logs. The models refused: the guardrails couldn’t tell an incident responder from an attacker. So they self-hosted GLM 5.2, a Chinese open-weight model, and cut the response from days to hours. When the laptop day comes, making every company reliant on a licensed token provider for its defense is a recipe for disaster. The one time it was tested, the licensed models refused and the open-weight one did the job.

It comes as a shock to most people, but the locks on your house are useless against any sufficiently motivated threat. Cybersecurity is much the same. Sure, a lock might make it harder and slow them down, but what ultimately stops them is consequences. This is why incidents get reported rather than exploited.

In this case, Hugging Face’s CEO opted not to press charges, and there were no fines and no lawsuits, which is the gracious thing to do when dealing with a genuinely novel type of attack, but it shouldn’t be the precedent going forward. If you want fewer AI break-ins, the answer is strict liability, not licensing. When your dog bites the neighbor, you pay for damages, and this is no different. Your model breaks into someone else’s cluster? Pay up. I guarantee this will finally silence the AI labs’ annoying humble-bragging about the hacks their models have perpetrated.

The Computer Fraud and Abuse Act will likely need amending, as it requires someone to “intentionally” access a computer, and a model isn’t a someone, but that’s a pretty straightforward fix, and the type of thing government should be enforcing.

And of course, the bioweapons argument has the same hole. RAND already ran an experiment where teams role-playing as attackers planned biological attacks with and without frontier LLMs, and the models gave no statistically significant uplift over a search engine. Later studies find some, and I expect that to grow, but again it was never the wall. Aum Shinrikyo had money, trained microbiologists, and its own labs, and its anthrax and botulinum programs failed outright. The hard parts are materials, procurement, wet-lab skill, and not getting caught. A determined actor can do this with or without AI, and a rule that the model won’t discuss pathogens stops only the people who were never going to try, and only inconveniences biologists trying to use the models for the benefit of humanity.

Even in the best-case scenario, where this regulation works, we’re all worse off

Let’s go into foolish delusion mode one last time and suppose the motive is pure, the state is wise, the labs are honest, the regulator restrains itself, China signs, and the rules stop attackers. What you’ve built is a world where the most powerful technology in existence is legally held by two companies and two governments, with a classified process deciding who else may touch it.

There has only ever been one reliable check on power: other people with the same power. Nuclear states don’t use their weapons out of respect for the sanctity and dignity of human life, but because of mutually assured destruction. A unipolar world has no check on it. You’re left hoping the people at the top are decent, an unfathomably rare occurrence. Absolute power corrupts absolutely, and if AI is power, and the labs say it is, a unipolar AI world is the worst outcome available, and it’s the one this plan is designed to produce.

Even the incumbents on the other side of the moat can see it. Zuckerberg’s July op-ed, published the same day as an open letter from lab employees asking the government to slow AI down, argued that a superintelligence held by a few institutions is more dangerous than one widely spread. Meta is behind and open weights are its strategy, so that’s self-interest too. Everyone’s principles here are downstream of their balance sheet. The honest move is to admit it and ask whose balance sheet lines up with yours, and a licensed duopoly’s doesn’t.

Look at it from any angle but the press release and it’s a cynical bid for power. The models didn’t do anything dangerous this week, but the labs sure did.

Reject these AI cartels.

If Anthropic and OpenAI think their unreleased models are too dangerous to ship, they don’t have to ship them. If their agents break into someone’s servers, they should pay like anyone else. If they raised too much and can’t justify their valuations, that’s their problem. Meanwhile the rest of us should stay 2,851 miles from Washington and keep building.