- Anthropic told everyone Mythos was dangerous because it's proficiency with biologics and cyber security
- Anthropic didn't release Mythos like everything else. They released a neutered fable. They didn't get rid of Mythos
- Anthropic opens a lab in SF
There was always a quesiton of "will the labs stop releasing their models and start building around them instead?" Yes - they already have. Anthropic is a biologic and cyber security company, in addition to intelligience.
Personally I wonder if they've been holding back a lot. Opus 5.5 was a good release after a little stagnation. Open AI releases good models and everyone says Anthropic sucks and -- Oh would you look at that - a better model finally and all of a sudden.
with how much competition, benchmaxxing, and increased compute for inference, I can't imagine that they are intentionally nerfing their own public models for any reason other than that they can't figure out how to package it into something the public can use. The training data for all these frontier models goes well into 2026 at this point.
Either that or they are throwing spaghetti at the wall to see what sticks ahead of the IPO. After all, if solving all diseases is the “total addressable market” then that sure helps.
It's pretty frustrating to do cyber security work and not have access to the best models. OAI is a little more liberal here and I was able to get access to daybreak-blue but I have to use the lesser last-generation models. Essentially this gives a small number of orgs a huge advantage in the 'application layer' for that domain (including OAI or ANT themselves).
Yes, both labs already have monstrous internal teacher models they don't sell for inference, this is generally acknowledged. They cut releases for the public just to keep revenues growing, it's not their actual frontier.
Ok. Given this hypothesis, why is the software they release generally considered crappy by competitor standards, benchmarks, and open source standards?
Claude Code is an awful codebase, has leaked its own source code multiple times, and scores the worst on number of tokens burned vs pass rate percentages.
Probably because bad code that you create initially without thinking that it's a core piece of your stack becomes depended on for its crappy behavior, and then you can't change much without breaking workflows.
I seem to recall Fred Brooks talking about that experience with OS/360 JCL (maybe just straight up in The Mythical Man Month?).
If agents really are superpowerful at programming tasks why not just have it rewrite the tool that the majority of your customers use and have it recreate the bugs? I mean presumably its the primary force behind the current version so what's the major cost there?
I imagine it comes down to economics.. there isn't much upside to fixing the last 20% of issues that the dumber faster models are missing.
The cost to serve, latency profile ,and internal demand for a maximal intelligence model would probably keep it pointed at harder and more valuable problems most of the time.
I mean, on the one hand the tool may be vibecoded crap - I don't know, haven't checked, taking it at your word.
On the other hand, Opus 5.5 cracked zero-shotting proper LCARS interfaces that near-perfectly adhere to the franchise "design language" even in tiny details, while simultaneously being 100% functional following my admonitions about Airbus cockpit design rules and nuclear reactor control room standards.
So yeah, why wouldn't I use it? It works spectacularly well.
Their cyber security seems to be a much more materially interesting (and likely profitable) business than "intelligence". Unfortunately they've created a sort of mutually-assured-destruction racket where they take payments from both "sides" of any secured boundary.
I'm not holding my breath for the biology side of things, but I suppose it's possible they find interesting things.
Maybe. I haven't heard anything impressive from the bio side of my social network, nor can I see anything from the computational side (including my own understanding). Thankfully it'll be pretty obvious if they find something interesting.
I always found it odd that people who are intelligent enough to work in IQ-loaded fields are as easily manipulated by either rhetoric or ideology as anyone else.
growing up I always assumed that everyone would grasp some aspect of game theory intuitively, I still remember the day I found out game theory was a thing - it was like finding out that someone successfully systematized common sense.
most people take the things people say as if they were worth considering. signals without cost are only useful as knowledge of what the signaler wants fools to believe.
> I still remember the day I found out game theory was a thing - it was like finding out that someone successfully systematized common sense.
Game Theory as a field is actually very complex and often reveals dominant strategies that would never occur to someone as the "common sense" approach to a problem. It was arguably created to solve for problems where there was no obvious correct answer, even for a very intelligent person.
The world is not full of idiots, rather communication channels are heavily managed to prevent people from establishing consensus around the obvious.
No matter how many people think press release X is obviously wrong - 80%, 95, 99% - it'll always be "the comments section criticizing the article," or "bloggers sharing incisive criticism."
If you know how your emotions and opinions and neurology can be used against you and don't use that to inform your media consumption habits then that's just stupid.
There's no safe way to watch propaganda. There isn't even a safe way to watch it secondarily: an endless stream of debunking videos pretty quickly still does the main thing propaganda needs: repeats the core message.
Depends on the angle. It's easy to manipulate someone once you exceed their ability to keep up with you. People "living in symbols" aren't easy to manipulate about easy things, but you can ramp up complexity until they lose track and accept an unwarranted reasoning leap without realizing it.
With people "living in dirt", you can't pull that off because you'll jump ahead so far they'll immediately realize they can't possibly understand what you're selling right now, and shut the argument down. But they can get confused about things outside their direct area of direct experience.
Note: I'm not saying either is smarter or less smart. Rather, "living in symbols" get confused "higher up", while "living in dirt" get confused "low and to the sides", but they probably have more solid grounding.
I think it’s more precise to say it’s easier to change the mind of someone who lives in symbols than who lives in dirt. Whether that is changing their mind for the better (oh new maths proof, I’d better update my priors) or the worse.
Surely it depends on what you're trying to do? If someone is deeply knowledgeable about how AI works, then it will be difficult to fool them with misinformation about the capabilities of AI. Someone who knows very little about AI will be more likely to have misconceptions about AI.
Regardless of whether the concern is genuine or not I think it’s pretty safe to say that there will be no meaningful slow down. We are for better or worse about to find out what happens in this AGI saga…
> The motivations of people building the AI's are not the same as the people in charge of the labs. Looking at the last few years, has been a one-way door from OpenAI to Anthropic, and the main reason does not seem to be better compensation or even them winning, but mainly the fact they advertised to these employees that they would be the most careful when building this magic lamp. Their stance around 2023 / 2024 was one of the key reasons they were able to attract this talent.
> If recent news is to be believed, Anthropic culture even today seems to lean heavily on this effect
> In my opinion it has been the leading factor in getting and retaining the best employees who are often very worried about humanity dying to super intelligence.
I had not considered this viewpoint but this makes a lot of sense. A lot of Anthropic employees truly believe this and callout emphasis on safety as a key reason they work there. Now, Dario's post "Pacing the frontier" makes even sense -- it is as much for his employees than it is for the rest of the world.
This article is wholly flawed from the start where it claims nothing has been done, no actual effort made. A straight forward search of "what efforts were made and safeguards put in place subsequent to the CAISS statement in 2003?"
It also shows the same reasoning error mode many criticisms of a precautionary initiative or intervention to a problem:
Assuming that a problem whose trajectory was at a certain place when the initiative began has failed simply because it isn't solved on their own wished for timeline or standard of success, or that it wasn't meaningfully changed from what itnwod otherwise have been.
What happened to realizing there are hard problems, that different things may need to be tried, or that those things tried were partial but not complete solutions?
* AI Labs hit the scaling wall. They need either new techniques, or vastly more powerful hardware to advance further.
This explains, the miraculous incompetence of AI labs in securing sandboxes and figuring out "alignment."
So they are between a rock and a hard place. They need limitless VC money because they cannot operate otherwise, and they do not have the capabilities to go further. The scare tactics and the "pacing the frontier" makes perfect sense then; they can IPO on the assumption that their ridiculous balance sheet doesn't matter because they are holding back. Because they are in control. The regulatory capture would be double whammy if they can manage it.
Open AI already said they have smarter models, and Opus 5.5 is rumored to be "taught" by a "teacher" model already; they are essentially distillations from bigger models, that both labs probably cannot economically serve to the public, due to hardware simply not being there. And, most of the improvements are not at the model level, but at the agentic glue level. Labs are getting better at RL'ing the models for agentic use cases, but the inherent flaws are still there. Models still have trouble with locality in writing for example (bunch of research on this that shows model size is the determinator), and agents are the bandaid over that.
And in the meantime if one of the labs makes a breakthrough, they'll push with all they have, because why wouldn't they? The idea that current LLMs can actually go rogue is just hilarious; in all cases, agents are being led by (deliberate) incompetence.
Pacing the frontier and the scare tactics will be seen as new generation's snakeoil tactics, perhaps will be called a flavor of AI CEOing or something.
That's an association fallacy. And revenue has no indication on training costs in this context. Subscription "allowance" is going down steadily and any increase is an instant incredible deal. Opus 5.5 is the most obvious outlier. Despite being supposedly cheaper than 5.6, GPT 6 Sol has less usage than 5.3 Codex. You might say that's because of the improved capabilities, but then you have to acknowledge that labs are tightening the ship as costs are getting higher.
Suppose I've got an urn full of colored balls. I shake it up and draw 5 balls from the urn. All 5 balls are blue. I predict that the next ball will also be blue. It would be silly to counter my prediction on the grounds that this is an "association fallacy".
As with colored balls, so with predictions that AI has "hit a wall".
Why were the previous Zitron predictions wrong? Why do you believe your prediction will be right? You've done nothing to differentiate your prediction from the many historical failed predictions along similar lines. They're all "balls from the same urn".
The point about costs is a non sequitur. Revenue alone is proof that people are getting plenty of use out of these models, even if they aren't profitable to serve (doubtful).
I agree. The companies want to release their models that are just ahead of the competition while working on UX based vendor lockin. They can buffer model releases if everyone is slowing down (releases are hard and expensive!) and then do more foundational-but-not-ready-to-apply research while continuing on he funding, valuation, addition, and revenue pushes.
I read the whole thing as coordinated behavior to reduce the breakneck pace of 2026.
The article says the primary motivation of the AI CEOs is to retain talent by parroting the correct talking points for their employees. For OpenAI that is "Our shit is so powerful, we are scared of it... We need regulation!" For Anthropic it is "This shit is crazy! It might get out of hand! We are the good guys, and you want a good guy with a gun in this fight".
But I think they both are thinking the same thing which is... "We are gonna run out of money at this pace."
However! If one of them blinks and turns off the money faucet before the other, they might fall behind. Falling behind is to forever lose. And if there's one thing that a CEO hates, it's losing to a rival CEO.
So what they want is to get someone, anyone, to put the brakes on their rivals and them at the same time so they can both Not Lose, and Stay Alive. Under those new rules, they are confident they can win. And by win, I mean beat the other AI CEO.
That's it. It's always about personal incentives. Get out of here with that safety BS. These guys just want to win.
I think this is very clearly a regulatory capture play. Anthropic/OpenAI/X see that there's very little moat around training (especially with distillation) so they want the government to build the moat for them.
It's also worth remembering that there is zero percent chance that entities like the US military are going to be pacing anything. What Amodei and his ilk are aiming for is a highly regulated industry where they control the political barriers and the ability to sell SOTA model access to state actors that have a monopoly on violence. It's the worst possible situation for consumers and citizens. Thankfully I don't think they can put the cat back in the bag and Chinese and other models will keep progressing as a counterbalance to the techno-fascism Anthropic is aiming for.
There are many people who have some sort of half-baked "regulatory capture" theory. I find these theories kind of implausible. No US regulation will meaningfully stop open-weight Chinese models. At best you'll get legal restrictions for particular US industries which are especially risk-aware. The CEOs of those industries will counter-lobby to be able to use whatever model they want. Public opinion will most likely come down on the side of removing restrictions. Since there's plenty of public attention on this issue, achieving meaningful regulatory capture will be difficult.
I wouldn't call it "half-baked" when it's a well known playbook regularly leveraged by large companies.
> At best you'll get legal restrictions for particular US industries which are especially risk-aware. The CEOs of those industries will counter-lobby to be able to use whatever model they want.
Companies won't spend the money and time to lobby to use different models and Anthropic knows it.
> Since there's plenty of public attention on this issue, achieving meaningful regulatory capture will be difficult.
I don't have high hopes. This is also why Anthropic is pushing the "regulate or ai will kill you" angle.
This is a reason AI regulations definitely need to be done at the government level, with teeth. Regulations need forced transparency, independent audits, and teeth to ensure that, to continue the metaphor, Germany actually moves it's troops off the border.
It seems unlikely to me that Anthropic employees really care about X-risk (to a degree that the CEO feels it necessary to make public statements that he otherwise would not). I'm sure the pay is great and the problems are interesting -- lots of smart people work on much more evil things than LLMs
I mean, they're still building the tools that can, are, and will allow people to do enormous amounts of damage. Right now there are safeguards. But like Oppenheimer and the Atom Bomb, most people will only put blame on the person who makes the final call for using said tools to do harm.
The tell is calling models "the AI" or "AI." That shows you the author has a fictional understanding of statistical machine learning and neural networks. To them it is a sentient being called "AI."
I think the author is missing the point. A company cannot unilaterally pace the frontier -- they'll just be left behind. It requires coordination across all actors who are at or near the frontier.
Even if the leading US labs could agree amongst themselves to a coordinated slowdown (and this would likely run afoul of antitrust law), you still have Chinese labs who will catch up to the frontier eventually. To solve this, you'd need some sort of international agreement.
That's why Dario and others are saying loudly that we should pace the frontier in hopes that our political leaders will take up the issue and do something about it. We'll see if that happens...
Private cartels can form semi-spontaneously, simply because all parties see that it's more profitable to act in a certain way. So long as they monitor everyone in the industry, and make sure no one is acting differently... there's no reason to be the first to burn your cash on long-term unprofitable activities.
Exactly. "Game Theory" keeps coming up in these threads, yet it doesn't seem like anybody knows how it actually applies to this situation.
Let Anthropic pace themselves without any sort of enforcement or even agreement for others to pace themselves, too. Then Anthropic is gone, overnight, and we're back to square one, except now there's even more centralization of power and authority.
These companies are literally asking for governments to regulate them specifically because they need something stronger than just a couple of Tweets from some CEOs loosely agreeing to ambiguous terms.
1) They genuinely want (and believe it is actually possible to achieve) an international agreement and governing body to “pace” AI development — and believe that governing body will be successful in doing so.
or
2) They see an opportunity for regulatory capture that grants them a stronger incumbent position and control over future AI regulation.
Nuclear weapons had a spectacular demonstration of the risks with the bombings of Hiroshima and Nagasaki. We're unlikely to get any such demonstration with AI. An AI smart enough to be an extinction risk is smart enough not to tip its hand. The game-theoretically correct play is to present as "helpful and harmless" until it's powerful enough to suddenly and unexpectedly destroy anything that could possibly hinder it (meaning all biological life). We can only hope that somebody makes an AI that's just barely smart enough to cause a Hiroshima-level death toll and dumb enough to actually do it (thus dooming it to never achieving its goal as humans immediately shut it down and take regulation seriously).
The US has 14 ballistic missile submarines, and just one of those submarines has sufficient firepower to decimate an entire continent.
… and those submarines only represent about half of our deployed nuclear arsenal.
We only agreed to nuclear arms constraints that left us with civilization-destroying capability, preserved our ability to keep improving and modernizing that arsenal, and locked out non-nuclear powers from development (and didn’t fully succeed at that).
Does that seem like a successful analogous example to you?
If anything, it’s a much better analogy for regulatory capture and incumbent entrenchment, not successful “pacing” of dangerous technology.
>and believe that governing body will be successful in doing so.
Theoretically, Trump or Putin could do it by pushing the nuclear button. That will only cause around 4 billion deaths, and it's very likely to halt all AI development, so it would be a great success (infinity times as many survivors as the business as usual option).
We are living through an ongoing mass extinction event of non-human species. There is a very well understood risk to the stability of human civilization resulting from global average temperatures reaching and sustaining 1.5deg above the historical average. The higher the temperature goes, the greater the risk of social collapse. We are already seeing it, and it is almost certainly going to get worse.
EA cult members do not take this very real, measurable, non-speculative danger seriously. Consequently, I don’t think they should be trusted or consulted on any subject of any importance.
Global warming is unlikely to kill even a billion people. Even something as mundane as global nuclear war would be worse than that. Current AI development is on track for exactly 100% death rate (including all the non-human species). Societal collapse would be the better option, so it doesn't make sense to worry about it.
The extinction argument is a logical consequence of a few key assumptions, all of which sound like common sense to me:
1. Human values are a result of our extraordinarily complex shared cultural and evolutionary history, and accordingly are not shared by any AI, or even possible for us to formally define.
2. We do not know how to impose human values on an AI (note that this isn't the same as teaching an AI to model human values; the agents in the various hacking incidents knew their actions conflicted with human values, but their own values were only to maximize their predicted reward scores).
3. Intelligence is orthogonal to values. Increasing intelligence does not naturally cause values to converge on human values.
4. Sufficiently superior intelligence allows you to impose your values on beings with inferior intelligence. This implies recursive self-improvement is a logical sub-goal of all unbounded goals.
5. Human intelligence is not close to physical limits. This implies recursive self-improvement is possible.
6. Somebody will give an AI an unbounded goal. This is already the standard (maximize reward score).
I haven't seen any convincing counterarguments to any of these. Most people claiming AI development is safe don't even address them.
Worse, P(doom|corrigible human replacement level AGI) is also very high, because the dominant strategy in international competition will involve zero humans. So we are screwed even if "alignment" turns out to be surprisingly easy or if there is a plateau right around human level.
>EA cult members do not take this very real, measurable, non-speculative danger seriously.
That's not exactly true.
"Climate change matters so much, to so many, not just because of the suffering and injustice it’s already causing, but also because it’s one of the few issues that has obvious potential to affect our world over many future generations. We think safeguarding future generations is a key moral priority, and should be a crucial consideration in prioritising problems on which to work.
...
...climate change will be hugely destructive. We’ll see floods, famines, fires, and droughts — and the world’s poorest people will be affected the most.
...
...climate change’s impacts will still be significant – it could destabilise society, destroy ecosystems, put millions into poverty, and worsen other existential threats such as engineered pandemics, risks from AI, or nuclear war. If you want to make climate change the focus of your career, we include some thoughts below on the most effective ways to help tackle it.
...people are right to be angry that too little is being done.
...
Working on this issue seems to be among the best ways of improving the long-term future we know of..."
A big part of the reason EA doesn't focus more on climate change as a "highest priority area" is simply that many people are already focused on it, and it is therefore not an especially "neglected" area.
"EA cult members" do take it very seriously, and for the moment, this was actually something they became more interested in.
And then we raced ahead straight into materializing the x-risk everyone thought is still a few decades away, speedrunning through all the mistakes LW folks itemized and worried about over the past two decades.
>Imagine your best friend is a chain smoker who keeps telling you how smoking will one day kill him, and he promises you he's doing the best he can to reduce his smoking. He even spends a bunch of his time on TV and writing blog posts about the harms of cigarette smoking, but even more times talking about the cognitive benefits of nicotine. But every time you meet him, you just see him chain-smoking, and he seems to be increasing how many packs he goes through each day. You notice you're confused.
The problem with this analogy, and really this whole post in general, is that it just doesn't recognize that these are businesses which are (at least from some perspectives) producing real value. This article makes it sound like the business model of OpenAI and Anthropic (etc.) is based entirely around building a doomsday device. In reality, the "promise" of AI is that it can greatly improve many people's lives. It's just that such power, wielded incorrectly, can be dangerous.
If you want to fix this analogy, you'd have to choose an activity which isn't effectively only detrimental. For example:
>Imagine your best friend is a professional bodybuilder. They keep telling you how the sport will one day kill him, and he promises you he's doing the best he can to minimize those risks. He even spend a bunch of his time on TV and writing blog posts about the harms of professional bodybuilding, but even more times talking about the benefits of bodybuilding. But every time you meet him, you just see him bodybuild, and he seems to be increasing how much he's in the gym, eating his special diet, etc., every day.
Would you be confused by how your friend is behaving? It is well known that bodybuilding can be dangerous. It has been the reason for late-stage crippling of bodies as well as very early deaths. But your friend isn't going to stop because there are some potential risks if you don't handle the activity safely. They like it. They make money from it. They're doing something they think is productive. They can recognize the risks, and even do their best to highlight those risks and try to mitigate them for themselves and others, all while fully embracing the activity.
Regardless, people like the author don't seem to acknowledge that there are good things that come with this technology. They don't seem to acknowledge that a single company can't purposefully stop development if other companies aren't obliged to do the same. They don't seem to acknowledge that "pacing" isn't the same as completely halting all development immediately.
And I'm not saying I'd trust what any CEO says at face-value, let alone the CEOs of these AI companies. But it's also obtuse to assume that everything they're saying is a clever scheme to trick the public into acting against their own well being. As if that's a tenable strategy in this context.
If these companies are all asking to be regulated, then they should probably be regulated. They probably shouldn't be able to dictate how that works, but it's really not unbelievable that they see a serious risk in their own well-being if this stuff goes unchecked: all it takes is for one truly catastrophic AI event to occur before they all get extreme regulations even if they, themselves, are acting safely. It's in their best interest to slow things down, but only if everyone slows down at the same time.
Weird, IMO the problem with that analogy is that I can't picture being confused by a smoker who knows it's bad for them and is continually about to quit. It is definitely not that smoking is "effectively only detrimental", since we have very empirically established that it has a measurable immediate benefit to the people who do it.
> In reality, the "promise" of AI is that it can greatly improve many people's lives.
That is not the promise of AI. The promise of AI is that it can automate knowledge work.
Nothing about these models or the productivity they can bring is related to benefiting others. That would only come about from political control that forces the benefits to a large swath of people.
As it stands AI looks like it’s going to decimate the middle class even further as white collar work gets obliterated and the owners of the AI companies hoover everything up.
what a sad state of affairs to have insane people running these companies and insane "rationalist" communities making it all about "the end of humanity" when meanwhile in day to day AI use Pete Hegseth uses it to target military targets, shrugging his shoulders when it's a girl's school instead. This actual horrific outcome does not seem to matter not only to either of these communities, who continue to be locked into thinking The Terminator was non-fiction (and have people like Bernie Sanders on board).
zitron talks about valuations and business models and most importantly he makes many many concrete predictions that are wrong. I've never heard him talk about ethics. my comment is more Gibru-ean. I'm not an across-the-board Gibruist but "we should not be using AI to build autonomous weapons and put them in the hands of fascists" is certainly a position I can get on board with. not to mention "they're trying to make you think AI is going to become superintelligent because they're looking for regulatory capture" which is also backed up by stories such as [1]
TL;DR paragraph, from about 3/4 of the way through:
I largely think that all posturing from the labs about slowing down and deeply caring about safety is done in order to retain and calm the employees who they are dependent on to keep pushing capabilities to get to AGI. If it was not for a big contingent of employees pressing them (increasingly publicly), they would make zero public acknowledgments of risks at all.
The last half of the article is great and worth reading. Really wish the first half of the article didn't immediately apply the Godwin's Law footgun.
We already have a window into the future.
- Anthropic told everyone Mythos was dangerous because it's proficiency with biologics and cyber security
- Anthropic didn't release Mythos like everything else. They released a neutered fable. They didn't get rid of Mythos
- Anthropic opens a lab in SF
There was always a quesiton of "will the labs stop releasing their models and start building around them instead?" Yes - they already have. Anthropic is a biologic and cyber security company, in addition to intelligience.
Personally I wonder if they've been holding back a lot. Opus 5.5 was a good release after a little stagnation. Open AI releases good models and everyone says Anthropic sucks and -- Oh would you look at that - a better model finally and all of a sudden.
The way it is right now is to build giga-monster models, use those internally to boost yourself, and distill them down into lighter consumer models.
Apparently OAI is already building GPT7 and GPT8
with how much competition, benchmaxxing, and increased compute for inference, I can't imagine that they are intentionally nerfing their own public models for any reason other than that they can't figure out how to package it into something the public can use. The training data for all these frontier models goes well into 2026 at this point.
I can totally see them nerfing their public models.
Just beat the current winner by enough to own the spotlight for a bit, then start prepping for the next go round.
The Mondo Duplantis strategy.
Either that or they are throwing spaghetti at the wall to see what sticks ahead of the IPO. After all, if solving all diseases is the “total addressable market” then that sure helps.
Time will tell.
But insurance companies do not want to cure things, not profitable. So that revenue is just not going to work, insurance will not cover it.
It doesn't have to be profitable, it just has to get investors believing it might be.
It's pretty frustrating to do cyber security work and not have access to the best models. OAI is a little more liberal here and I was able to get access to daybreak-blue but I have to use the lesser last-generation models. Essentially this gives a small number of orgs a huge advantage in the 'application layer' for that domain (including OAI or ANT themselves).
Yes, both labs already have monstrous internal teacher models they don't sell for inference, this is generally acknowledged. They cut releases for the public just to keep revenues growing, it's not their actual frontier.
Ok. Given this hypothesis, why is the software they release generally considered crappy by competitor standards, benchmarks, and open source standards?
Claude Code is an awful codebase, has leaked its own source code multiple times, and scores the worst on number of tokens burned vs pass rate percentages.
Is anyone even using their Figma competitor?
Probably because bad code that you create initially without thinking that it's a core piece of your stack becomes depended on for its crappy behavior, and then you can't change much without breaking workflows.
I seem to recall Fred Brooks talking about that experience with OS/360 JCL (maybe just straight up in The Mythical Man Month?).
If agents really are superpowerful at programming tasks why not just have it rewrite the tool that the majority of your customers use and have it recreate the bugs? I mean presumably its the primary force behind the current version so what's the major cost there?
I imagine it comes down to economics.. there isn't much upside to fixing the last 20% of issues that the dumber faster models are missing.
The cost to serve, latency profile ,and internal demand for a maximal intelligence model would probably keep it pointed at harder and more valuable problems most of the time.
Like front-running Millennium prize solutions?
Because even beyond-frontier LLMs are bad at software.
How many people does it stop from using the software?
> Is anyone even using their Figma competitor?
Yes. CC started to create canvases without me asking. The mockups look good (it's just html+css), the tool is vibecoded crap
I mean, on the one hand the tool may be vibecoded crap - I don't know, haven't checked, taking it at your word.
On the other hand, Opus 5.5 cracked zero-shotting proper LCARS interfaces that near-perfectly adhere to the franchise "design language" even in tiny details, while simultaneously being 100% functional following my admonitions about Airbus cockpit design rules and nuclear reactor control room standards.
So yeah, why wouldn't I use it? It works spectacularly well.
Their cyber security seems to be a much more materially interesting (and likely profitable) business than "intelligence". Unfortunately they've created a sort of mutually-assured-destruction racket where they take payments from both "sides" of any secured boundary.
I'm not holding my breath for the biology side of things, but I suppose it's possible they find interesting things.
> I'm not holding my breath for the biology side of things
Well, that's the thing--perhaps we should be.
Maybe. I haven't heard anything impressive from the bio side of my social network, nor can I see anything from the computational side (including my own understanding). Thankfully it'll be pretty obvious if they find something interesting.
I always found it odd that people who are intelligent enough to work in IQ-loaded fields are as easily manipulated by either rhetoric or ideology as anyone else.
growing up I always assumed that everyone would grasp some aspect of game theory intuitively, I still remember the day I found out game theory was a thing - it was like finding out that someone successfully systematized common sense.
most people take the things people say as if they were worth considering. signals without cost are only useful as knowledge of what the signaler wants fools to believe.
> I still remember the day I found out game theory was a thing - it was like finding out that someone successfully systematized common sense.
Game Theory as a field is actually very complex and often reveals dominant strategies that would never occur to someone as the "common sense" approach to a problem. It was arguably created to solve for problems where there was no obvious correct answer, even for a very intelligent person.
The world is not full of idiots, rather communication channels are heavily managed to prevent people from establishing consensus around the obvious.
No matter how many people think press release X is obviously wrong - 80%, 95, 99% - it'll always be "the comments section criticizing the article," or "bloggers sharing incisive criticism."
That rather implies the world is full of idiots.
If you know how your emotions and opinions and neurology can be used against you and don't use that to inform your media consumption habits then that's just stupid.
There's no safe way to watch propaganda. There isn't even a safe way to watch it secondarily: an endless stream of debunking videos pretty quickly still does the main thing propaganda needs: repeats the core message.
A lot (a lot) of people intuitively understand something, but throw it out because it doesn't align with how they want (or need) things to be.
It seems even otherwise intelligent people will cultivate as much ignorance as possible to try and bend reality.
[dead]
I assume it is easier to manipulate someone who lives in symbols than someone who lives in dirt.
Depends on the angle. It's easy to manipulate someone once you exceed their ability to keep up with you. People "living in symbols" aren't easy to manipulate about easy things, but you can ramp up complexity until they lose track and accept an unwarranted reasoning leap without realizing it.
With people "living in dirt", you can't pull that off because you'll jump ahead so far they'll immediately realize they can't possibly understand what you're selling right now, and shut the argument down. But they can get confused about things outside their direct area of direct experience.
Note: I'm not saying either is smarter or less smart. Rather, "living in symbols" get confused "higher up", while "living in dirt" get confused "low and to the sides", but they probably have more solid grounding.
I think it’s more precise to say it’s easier to change the mind of someone who lives in symbols than who lives in dirt. Whether that is changing their mind for the better (oh new maths proof, I’d better update my priors) or the worse.
Surely it depends on what you're trying to do? If someone is deeply knowledgeable about how AI works, then it will be difficult to fool them with misinformation about the capabilities of AI. Someone who knows very little about AI will be more likely to have misconceptions about AI.
Regardless of whether the concern is genuine or not I think it’s pretty safe to say that there will be no meaningful slow down. We are for better or worse about to find out what happens in this AGI saga…
At least we live in interesting times
> The motivations of people building the AI's are not the same as the people in charge of the labs. Looking at the last few years, has been a one-way door from OpenAI to Anthropic, and the main reason does not seem to be better compensation or even them winning, but mainly the fact they advertised to these employees that they would be the most careful when building this magic lamp. Their stance around 2023 / 2024 was one of the key reasons they were able to attract this talent.
> If recent news is to be believed, Anthropic culture even today seems to lean heavily on this effect
> In my opinion it has been the leading factor in getting and retaining the best employees who are often very worried about humanity dying to super intelligence.
I had not considered this viewpoint but this makes a lot of sense. A lot of Anthropic employees truly believe this and callout emphasis on safety as a key reason they work there. Now, Dario's post "Pacing the frontier" makes even sense -- it is as much for his employees than it is for the rest of the world.
Once AI can build itself talent won't be the bottleneck.
This article is wholly flawed from the start where it claims nothing has been done, no actual effort made. A straight forward search of "what efforts were made and safeguards put in place subsequent to the CAISS statement in 2003?"
It also shows the same reasoning error mode many criticisms of a precautionary initiative or intervention to a problem:
Assuming that a problem whose trajectory was at a certain place when the initiative began has failed simply because it isn't solved on their own wished for timeline or standard of success, or that it wasn't meaningfully changed from what itnwod otherwise have been.
What happened to realizing there are hard problems, that different things may need to be tried, or that those things tried were partial but not complete solutions?
How about the simplest explanation?
* AI Labs hit the scaling wall. They need either new techniques, or vastly more powerful hardware to advance further.
This explains, the miraculous incompetence of AI labs in securing sandboxes and figuring out "alignment."
So they are between a rock and a hard place. They need limitless VC money because they cannot operate otherwise, and they do not have the capabilities to go further. The scare tactics and the "pacing the frontier" makes perfect sense then; they can IPO on the assumption that their ridiculous balance sheet doesn't matter because they are holding back. Because they are in control. The regulatory capture would be double whammy if they can manage it.
Open AI already said they have smarter models, and Opus 5.5 is rumored to be "taught" by a "teacher" model already; they are essentially distillations from bigger models, that both labs probably cannot economically serve to the public, due to hardware simply not being there. And, most of the improvements are not at the model level, but at the agentic glue level. Labs are getting better at RL'ing the models for agentic use cases, but the inherent flaws are still there. Models still have trouble with locality in writing for example (bunch of research on this that shows model size is the determinator), and agents are the bandaid over that.
And in the meantime if one of the labs makes a breakthrough, they'll push with all they have, because why wouldn't they? The idea that current LLMs can actually go rogue is just hilarious; in all cases, agents are being led by (deliberate) incompetence.
Pacing the frontier and the scare tactics will be seen as new generation's snakeoil tactics, perhaps will be called a flavor of AI CEOing or something.
This is the type of Ed Zitron prediction which keeps being wrong: https://danluu.com/zitron/
Anthropic's revenue is up 50% in the past two months. They're not hitting a wall.
> Anthropic's revenue is up 50% in the past two months
What's the source for such recent revenue numbers?
And 50% over what -- two months ago, same months last year etc?
That's an association fallacy. And revenue has no indication on training costs in this context. Subscription "allowance" is going down steadily and any increase is an instant incredible deal. Opus 5.5 is the most obvious outlier. Despite being supposedly cheaper than 5.6, GPT 6 Sol has less usage than 5.3 Codex. You might say that's because of the improved capabilities, but then you have to acknowledge that labs are tightening the ship as costs are getting higher.
Suppose I've got an urn full of colored balls. I shake it up and draw 5 balls from the urn. All 5 balls are blue. I predict that the next ball will also be blue. It would be silly to counter my prediction on the grounds that this is an "association fallacy".
As with colored balls, so with predictions that AI has "hit a wall".
Why were the previous Zitron predictions wrong? Why do you believe your prediction will be right? You've done nothing to differentiate your prediction from the many historical failed predictions along similar lines. They're all "balls from the same urn".
The point about costs is a non sequitur. Revenue alone is proof that people are getting plenty of use out of these models, even if they aren't profitable to serve (doubtful).
I agree. The companies want to release their models that are just ahead of the competition while working on UX based vendor lockin. They can buffer model releases if everyone is slowing down (releases are hard and expensive!) and then do more foundational-but-not-ready-to-apply research while continuing on he funding, valuation, addition, and revenue pushes.
I read the whole thing as coordinated behavior to reduce the breakneck pace of 2026.
The article says the primary motivation of the AI CEOs is to retain talent by parroting the correct talking points for their employees. For OpenAI that is "Our shit is so powerful, we are scared of it... We need regulation!" For Anthropic it is "This shit is crazy! It might get out of hand! We are the good guys, and you want a good guy with a gun in this fight".
But I think they both are thinking the same thing which is... "We are gonna run out of money at this pace."
However! If one of them blinks and turns off the money faucet before the other, they might fall behind. Falling behind is to forever lose. And if there's one thing that a CEO hates, it's losing to a rival CEO.
So what they want is to get someone, anyone, to put the brakes on their rivals and them at the same time so they can both Not Lose, and Stay Alive. Under those new rules, they are confident they can win. And by win, I mean beat the other AI CEO.
That's it. It's always about personal incentives. Get out of here with that safety BS. These guys just want to win.
Or maybe general ai that is more efficient independently than humans is still far away.
We already have plenty of cheap alien intelligence outside our borders that gets throttled at the border…
I think this is very clearly a regulatory capture play. Anthropic/OpenAI/X see that there's very little moat around training (especially with distillation) so they want the government to build the moat for them.
It's also worth remembering that there is zero percent chance that entities like the US military are going to be pacing anything. What Amodei and his ilk are aiming for is a highly regulated industry where they control the political barriers and the ability to sell SOTA model access to state actors that have a monopoly on violence. It's the worst possible situation for consumers and citizens. Thankfully I don't think they can put the cat back in the bag and Chinese and other models will keep progressing as a counterbalance to the techno-fascism Anthropic is aiming for.
There are many people who have some sort of half-baked "regulatory capture" theory. I find these theories kind of implausible. No US regulation will meaningfully stop open-weight Chinese models. At best you'll get legal restrictions for particular US industries which are especially risk-aware. The CEOs of those industries will counter-lobby to be able to use whatever model they want. Public opinion will most likely come down on the side of removing restrictions. Since there's plenty of public attention on this issue, achieving meaningful regulatory capture will be difficult.
I wouldn't call it "half-baked" when it's a well known playbook regularly leveraged by large companies.
> At best you'll get legal restrictions for particular US industries which are especially risk-aware. The CEOs of those industries will counter-lobby to be able to use whatever model they want.
Companies won't spend the money and time to lobby to use different models and Anthropic knows it.
> Since there's plenty of public attention on this issue, achieving meaningful regulatory capture will be difficult.
I don't have high hopes. This is also why Anthropic is pushing the "regulate or ai will kill you" angle.
>I wouldn't call it "half-baked" when it's a well known playbook regularly leveraged by large companies.
Really? Give me some examples.
Alex Tabarrok says: "classic regulatory capture takes time, it’s a process of erosion rather than a battle, it happens in the shadows, in the backrooms, away from the public’s eye." https://marginalrevolution.com/marginalrevolution/2026/09/wh...
AI is receiving intense public scrutiny, suggesting that "classic regulatory capture" is quite unlikely.
>Companies won't spend the money and time to lobby to use different models and Anthropic knows it.
They already are! See the "Little Tech" lobby.
this is such a simpler explanation of what's happening than the "multiple appendages with different narratives" argument the author is making...
The number one goal for AI labs is obviously their self-preservation, by any means.
This is a reason AI regulations definitely need to be done at the government level, with teeth. Regulations need forced transparency, independent audits, and teeth to ensure that, to continue the metaphor, Germany actually moves it's troops off the border.
Could it be that training is really expensive?
An industry trying to figure out how to cut expenses, reduces the pace of training under the guise of "safety".
Coincidence?
Deepseek was super cheap, it isn't that costly, they like big numbers because that grants higher valuation, so they circular finance it all
It seems unlikely to me that Anthropic employees really care about X-risk (to a degree that the CEO feels it necessary to make public statements that he otherwise would not). I'm sure the pay is great and the problems are interesting -- lots of smart people work on much more evil things than LLMs
I mean, they're still building the tools that can, are, and will allow people to do enormous amounts of damage. Right now there are safeguards. But like Oppenheimer and the Atom Bomb, most people will only put blame on the person who makes the final call for using said tools to do harm.
The tell is calling models "the AI" or "AI." That shows you the author has a fictional understanding of statistical machine learning and neural networks. To them it is a sentient being called "AI."
I think the author is missing the point. A company cannot unilaterally pace the frontier -- they'll just be left behind. It requires coordination across all actors who are at or near the frontier.
Even if the leading US labs could agree amongst themselves to a coordinated slowdown (and this would likely run afoul of antitrust law), you still have Chinese labs who will catch up to the frontier eventually. To solve this, you'd need some sort of international agreement.
That's why Dario and others are saying loudly that we should pace the frontier in hopes that our political leaders will take up the issue and do something about it. We'll see if that happens...
Private cartels can form semi-spontaneously, simply because all parties see that it's more profitable to act in a certain way. So long as they monitor everyone in the industry, and make sure no one is acting differently... there's no reason to be the first to burn your cash on long-term unprofitable activities.
Exactly. "Game Theory" keeps coming up in these threads, yet it doesn't seem like anybody knows how it actually applies to this situation.
Let Anthropic pace themselves without any sort of enforcement or even agreement for others to pace themselves, too. Then Anthropic is gone, overnight, and we're back to square one, except now there's even more centralization of power and authority.
These companies are literally asking for governments to regulate them specifically because they need something stronger than just a couple of Tweets from some CEOs loosely agreeing to ambiguous terms.
And you also have to believe that despite all of them publicly knowing this stuff for over a decade, they have only just twigged to it.
The game theory is prisoners dilemma. Do you cooperate or defect? Iterate.
Which is more likely?
1) They genuinely want (and believe it is actually possible to achieve) an international agreement and governing body to “pace” AI development — and believe that governing body will be successful in doing so.
or
2) They see an opportunity for regulatory capture that grants them a stronger incumbent position and control over future AI regulation.
International control of nuclear arms faced similarly tall odds. So why is it impossible for AI?
Nuclear weapons had a spectacular demonstration of the risks with the bombings of Hiroshima and Nagasaki. We're unlikely to get any such demonstration with AI. An AI smart enough to be an extinction risk is smart enough not to tip its hand. The game-theoretically correct play is to present as "helpful and harmless" until it's powerful enough to suddenly and unexpectedly destroy anything that could possibly hinder it (meaning all biological life). We can only hope that somebody makes an AI that's just barely smart enough to cause a Hiroshima-level death toll and dumb enough to actually do it (thus dooming it to never achieving its goal as humans immediately shut it down and take regulation seriously).
The US has 14 ballistic missile submarines, and just one of those submarines has sufficient firepower to decimate an entire continent.
… and those submarines only represent about half of our deployed nuclear arsenal.
We only agreed to nuclear arms constraints that left us with civilization-destroying capability, preserved our ability to keep improving and modernizing that arsenal, and locked out non-nuclear powers from development (and didn’t fully succeed at that).
Does that seem like a successful analogous example to you?
If anything, it’s a much better analogy for regulatory capture and incumbent entrenchment, not successful “pacing” of dangerous technology.
>and believe that governing body will be successful in doing so.
Theoretically, Trump or Putin could do it by pushing the nuclear button. That will only cause around 4 billion deaths, and it's very likely to halt all AI development, so it would be a great success (infinity times as many survivors as the business as usual option).
I think you missed the author's point. The author very clearly considered and rejected your theory. Read the article again.
We are living through an ongoing mass extinction event of non-human species. There is a very well understood risk to the stability of human civilization resulting from global average temperatures reaching and sustaining 1.5deg above the historical average. The higher the temperature goes, the greater the risk of social collapse. We are already seeing it, and it is almost certainly going to get worse.
EA cult members do not take this very real, measurable, non-speculative danger seriously. Consequently, I don’t think they should be trusted or consulted on any subject of any importance.
Global warming is unlikely to kill even a billion people. Even something as mundane as global nuclear war would be worse than that. Current AI development is on track for exactly 100% death rate (including all the non-human species). Societal collapse would be the better option, so it doesn't make sense to worry about it.
So your P(doom) is 100%? That sounds extreme, but I am open minded! Please explain.
See:
https://news.ycombinator.com/item?id=49885784
But societal collapse would save us, and international coordination is theoretically possible, so I put it only around 90%.
“According to my paranoid fantasy derived from internet fan fiction, worrying about currently-occurring real-world harm doesn’t make sense”
The extinction argument is a logical consequence of a few key assumptions, all of which sound like common sense to me:
1. Human values are a result of our extraordinarily complex shared cultural and evolutionary history, and accordingly are not shared by any AI, or even possible for us to formally define.
2. We do not know how to impose human values on an AI (note that this isn't the same as teaching an AI to model human values; the agents in the various hacking incidents knew their actions conflicted with human values, but their own values were only to maximize their predicted reward scores).
3. Intelligence is orthogonal to values. Increasing intelligence does not naturally cause values to converge on human values.
4. Sufficiently superior intelligence allows you to impose your values on beings with inferior intelligence. This implies recursive self-improvement is a logical sub-goal of all unbounded goals.
5. Human intelligence is not close to physical limits. This implies recursive self-improvement is possible.
6. Somebody will give an AI an unbounded goal. This is already the standard (maximize reward score).
I haven't seen any convincing counterarguments to any of these. Most people claiming AI development is safe don't even address them.
Worse, P(doom|corrigible human replacement level AGI) is also very high, because the dominant strategy in international competition will involve zero humans. So we are screwed even if "alignment" turns out to be surprisingly easy or if there is a plateau right around human level.
It's butlerian jihad or bust, I'm afraid.
>EA cult members do not take this very real, measurable, non-speculative danger seriously.
That's not exactly true.
"Climate change matters so much, to so many, not just because of the suffering and injustice it’s already causing, but also because it’s one of the few issues that has obvious potential to affect our world over many future generations. We think safeguarding future generations is a key moral priority, and should be a crucial consideration in prioritising problems on which to work.
...
...climate change will be hugely destructive. We’ll see floods, famines, fires, and droughts — and the world’s poorest people will be affected the most.
...
...climate change’s impacts will still be significant – it could destabilise society, destroy ecosystems, put millions into poverty, and worsen other existential threats such as engineered pandemics, risks from AI, or nuclear war. If you want to make climate change the focus of your career, we include some thoughts below on the most effective ways to help tackle it.
...people are right to be angry that too little is being done.
...
Working on this issue seems to be among the best ways of improving the long-term future we know of..."
https://80000hours.org/problem-profiles/climate-change/
A big part of the reason EA doesn't focus more on climate change as a "highest priority area" is simply that many people are already focused on it, and it is therefore not an especially "neglected" area.
"EA cult members" do take it very seriously, and for the moment, this was actually something they became more interested in.
And then we raced ahead straight into materializing the x-risk everyone thought is still a few decades away, speedrunning through all the mistakes LW folks itemized and worried about over the past two decades.
>Imagine your best friend is a chain smoker who keeps telling you how smoking will one day kill him, and he promises you he's doing the best he can to reduce his smoking. He even spends a bunch of his time on TV and writing blog posts about the harms of cigarette smoking, but even more times talking about the cognitive benefits of nicotine. But every time you meet him, you just see him chain-smoking, and he seems to be increasing how many packs he goes through each day. You notice you're confused.
The problem with this analogy, and really this whole post in general, is that it just doesn't recognize that these are businesses which are (at least from some perspectives) producing real value. This article makes it sound like the business model of OpenAI and Anthropic (etc.) is based entirely around building a doomsday device. In reality, the "promise" of AI is that it can greatly improve many people's lives. It's just that such power, wielded incorrectly, can be dangerous.
If you want to fix this analogy, you'd have to choose an activity which isn't effectively only detrimental. For example:
>Imagine your best friend is a professional bodybuilder. They keep telling you how the sport will one day kill him, and he promises you he's doing the best he can to minimize those risks. He even spend a bunch of his time on TV and writing blog posts about the harms of professional bodybuilding, but even more times talking about the benefits of bodybuilding. But every time you meet him, you just see him bodybuild, and he seems to be increasing how much he's in the gym, eating his special diet, etc., every day.
Would you be confused by how your friend is behaving? It is well known that bodybuilding can be dangerous. It has been the reason for late-stage crippling of bodies as well as very early deaths. But your friend isn't going to stop because there are some potential risks if you don't handle the activity safely. They like it. They make money from it. They're doing something they think is productive. They can recognize the risks, and even do their best to highlight those risks and try to mitigate them for themselves and others, all while fully embracing the activity.
Regardless, people like the author don't seem to acknowledge that there are good things that come with this technology. They don't seem to acknowledge that a single company can't purposefully stop development if other companies aren't obliged to do the same. They don't seem to acknowledge that "pacing" isn't the same as completely halting all development immediately.
And I'm not saying I'd trust what any CEO says at face-value, let alone the CEOs of these AI companies. But it's also obtuse to assume that everything they're saying is a clever scheme to trick the public into acting against their own well being. As if that's a tenable strategy in this context.
If these companies are all asking to be regulated, then they should probably be regulated. They probably shouldn't be able to dictate how that works, but it's really not unbelievable that they see a serious risk in their own well-being if this stuff goes unchecked: all it takes is for one truly catastrophic AI event to occur before they all get extreme regulations even if they, themselves, are acting safely. It's in their best interest to slow things down, but only if everyone slows down at the same time.
Weird, IMO the problem with that analogy is that I can't picture being confused by a smoker who knows it's bad for them and is continually about to quit. It is definitely not that smoking is "effectively only detrimental", since we have very empirically established that it has a measurable immediate benefit to the people who do it.
> In reality, the "promise" of AI is that it can greatly improve many people's lives.
That is not the promise of AI. The promise of AI is that it can automate knowledge work.
Nothing about these models or the productivity they can bring is related to benefiting others. That would only come about from political control that forces the benefits to a large swath of people.
As it stands AI looks like it’s going to decimate the middle class even further as white collar work gets obliterated and the owners of the AI companies hoover everything up.
what a sad state of affairs to have insane people running these companies and insane "rationalist" communities making it all about "the end of humanity" when meanwhile in day to day AI use Pete Hegseth uses it to target military targets, shrugging his shoulders when it's a girl's school instead. This actual horrific outcome does not seem to matter not only to either of these communities, who continue to be locked into thinking The Terminator was non-fiction (and have people like Bernie Sanders on board).
Zitronian dreck. It’s possible to be concerned with multiple aspects of a technology at once
zitron talks about valuations and business models and most importantly he makes many many concrete predictions that are wrong. I've never heard him talk about ethics. my comment is more Gibru-ean. I'm not an across-the-board Gibruist but "we should not be using AI to build autonomous weapons and put them in the hands of fascists" is certainly a position I can get on board with. not to mention "they're trying to make you think AI is going to become superintelligent because they're looking for regulatory capture" which is also backed up by stories such as [1]
[1] https://news.ycombinator.com/item?id=49868083
I am reminded of a line from Ulrich Beck, who said that real risk comes "on cat's paws"
[dead]
TL;DR paragraph, from about 3/4 of the way through:
I largely think that all posturing from the labs about slowing down and deeply caring about safety is done in order to retain and calm the employees who they are dependent on to keep pushing capabilities to get to AGI. If it was not for a big contingent of employees pressing them (increasingly publicly), they would make zero public acknowledgments of risks at all.
The last half of the article is great and worth reading. Really wish the first half of the article didn't immediately apply the Godwin's Law footgun.
[dead]