Pet theory: 70% of what we're busy with brings no value outside of keeping people busy/employed, which means automating it and making it 100x faster also brings no value in the end, no matter how much "intelligence" you can deploy
An economist might counter that any such business would loose market share. But, in actuality, a lot busineses are not competing for survival or even for market share but have it due to entrenchment. examples include: ISPs, healthcare, etc
On the one hand this leads to the mediocre world we live in but on the other hand, thank goodness that these businesses exist. Imagine if every company were run like Tesla/SpaceX. You thought the horrendous birth rate in China was bad...it would be nothing compared to this theoretical world everybody is always in dog eat dog jungle survival mode.
I was reminded about how delusional people are when I was hanging out with some MAGA supporters in the south this weekend. These people adore Elon Musk and what he's done and the way that he runs his companies. They get this mentality because they work in government where their jobs are very cushy, but there is a lot of incompetence and bureaucracy going on, and they hate it. I honestly believe they wouldn't enjoy living in a world where Elon Musk mentality is running everything, including their jobs. That delusion makes them think that a world where Elon Musk is running everyone else's job is better for them.
I'd argue that of the people that Elon hires, many of them probably grew up in an environment where they had the support, love, and care that they need to become great because their predecessors worked in occupation that are not competing for survival. Else you end up with situations like China where the youths just don't have the ability (due to time, money and energy) to put down roots and build up the next generation. Their predecessors planted the seeds and nurtured them to grow value that is then extracted by Elon working the people into the ground and tossing them aside when they are no longer useful.
You can have cutthroat competition between companies and good working conditions, in fact more competition raises demand for labor so workers get more bargaining power. If employers are in survival mode they hold on to experienced workers at all costs and desperately try to attract new ones.
The problem in China comes from a concentrating economy and corrupt government that barely enforces labor laws. Demand for labor is not low but the number of employers is, so if you want better conditions and get rejected or fired you have nowhere else to go. America is on the same path.
>If employers are in survival mode they hold on to experienced workers at all costs and desperately try to attract new ones.
Can you give some examples? A company in survival mode can just as easily freeze hiring, cut benefits, and squeeze the people it already has. Needing experienced workers doesn’t mean it can afford to keep them.
Spirit is the example that bothers me. I’m biased because I loved flying with them, but former employees often describe good pay and working conditions alongside those cheap fares. How do you sustain that when costs rise and there’s barely any cushion for something going wrong? The big airlines have their credit card and loyalty businesses to lean on. What happens to a company trying to compete mainly on affordable flights?
My concern is that treating workers well costs money, and customers generally choose the cheaper or better product. What keeps a company that cuts corners from undercutting one that takes care of its employees?
I asked something similar at a HOPE talk recently about a worker owned software consultancy. Their answer was basically that everyone shares the pain when business gets rough. That sounds reasonable, and I want that model to work. I’m just not sure how well it scales.
>Demand for labor is not low but the number of employers is.
I also don’t think “too few employers” explains China very well. Look at the brutal competition among Chinese car companies, or in coffee, restaurants, and consumer goods. There can be plenty of competing businesses without workers having much leverage, especially when applicants outnumber the jobs available.
Weak labor law enforcement absolutely matters. But I don’t see why fierce competition and bad working conditions couldn’t reinforce each other. “We have to survive” can become the excuse for demanding more from everyone.
I want companies to improve their products while paying people well and treating them decently. I’m questioning whether competition alone gets us there. What makes employers compete for workers by improving conditions, instead of competing for customers by squeezing workers harder?
If you look at computer chess it improved steadily, there was a bit of a fuss when it beat Kasparov, then it kept going. People still play each other at chess much as before.
I figure it'll be like that with general intelligence - bit of as fuss as AI overtakes but humans will keep on humaning, making things, competing for status. Maybe like Jeeves and Wooster with us as Wooster and the AI as Jeeves. Most of their activities didn't bring much value apart from keeping people busy. AI will probably keep improving a few IQ points a year, not exploding but gradually getting pretty good.
For me the more interesting sci-fi is the merge / immortality stuff, curious how that goes.
Some sectors are less productive, but the reality is more mundane: only about 30% of the global workforce are knowledge workers. Adopting AI in real workflows takes time, and that time saved might not be used productively. Also solving Millennium problems is an amazing feat of engineering and mathematics, but doesn't change people's lives that much.
There is a name for this phenomena. It is called bullshit jobs[1]. The author David Graeber who coined the term argued that over half of jobs were bullshit jobs, so your 70% estimate is not far from what some actual scholars in the field also estimate.
I don't want to discount the amazing progress we've made with technology to improve people's lives.
There's the meme that the smartest people in the world are figuring out how to show ads. It's no different this time around. I understand that not everyone working on AI is building an ad network - but ultimately it feels like that's the end result.
It didn't feel quite this way 15 years ago. Back then, it felt much more tangible how technology was improving our lives. It is sort of tangible now too - I am doing things with AI that I otherwise couldn't. But somehow if feels like progress for the sake of progress.
And to be honest, it doesn't feel like the progress is for me or the people I care about.
> ... it doesn't feel like the progress is for me or the people I care about.
AI is doing very little to improve the human condition; all it seems to be good at is optimizing productivity and generating profits somewhere maybe. That is not an AI-generated problem; using profits as an all-things compass is a disease we have been cooking for ourselves for a long time. But I'm sort of hoping AI will make the problem so bad that we will be forced to fix it, or perish.
That’s one of the main points in Karp’s book. All the smart people are happy to spend all day pushing gooning content on children or triggering alcoholics to relapse, but ask them to tackle crime or defend the country and they’ll tell you it’s immoral.
Nobody tries to solve any actual problems anymore, they just produce meaningless consumer slop. And when people like Musk try to invent reusable rockets or electric cars, everyone says they’re stupid. And then they do it, and everyone says it was always going to happen anyway.
When is the last time Google or meta or whoever actually solved a problem? They’ll spend 500b in a year, enough to reforest the entire planet, and what do they actually give us? I couldn’t name a single thing they’ve done in the last ten years which actually produces value, but they’ve spent trillions.
It has exploded, the explosion just isn't evenly distributed. And just as it was with the internet, most people and even most industries are not actually using it yet. Many that are are using it in name only.
People can star github repos and signup their employees for a new subscription quickly, real adoption takes far longer.
Asking where is the intelligence explosion suffers the same issue as the Gorman Paradox[1] - the question proceeded the explosion by six months. Timing explosions is like timing the market - infinitely harder than predicting eventual existence.
Even though I started using LLMs as soon as they became available, I was until recently very cautious around using them autonomously. I was of the opinion that I needed to provide heavy supervision around coding tasks. Granted, that was probably justified based on the model capabilities at the time.
However, lately, I have had several experiences that have completely changed my mind - the here is a high level goal (involving gathering production logs, setting up an eval, running tests, iterating, including diagnosis and coming up with new ideas, etc. and don’t bother me until it’s done. The experience has been incredible.
This has been possible because the agent has a way to iterate and hillclimb against a metric it can measure. Relatively easy in the world of software.
I’m now convinced that we will unlock the same gains once make other domains similarly “iterable”.
Eg Agent comes up with 100 new proteins, runs in lab autonomously, gathers results, iterates. One month (or whatever) later, you have your new protein designed.
I think we're on the cusp of an intelligence explosion, but most jobs have no real purpose besides giving us sometime to do and a paycheck. Automating those jobs and making them 10x faster won't really help anyone. Read Bullshit Jobs by Graeber.
FWIW, why I'm rooting hard for AGI:
-In the best case, AGI will lead to UBI, so unemployment for all becomes the default scenario. Sounds scary but unemployment is better than many soul-sucking occupations, e.g. cold-calling.
-If AGI does result in a productivity boom due to automation, we're in for a deflationary spiral, so we don't need a huge amount of money for a decent standard of living.
It does lead to the question of how we'll find meaning in our lives, but I think we'll adapt in a few years.
As the entire universe gets converted into one gigantic paperclip, the last message written by humanity will probably be an article expressing skepticism whether AI progress is real or it's all just a marketing ploy.
Just like when you upscale a picture 100 times by 200%, and then you zoom in and start seeing all these artifacts, so is the outcome of "recursive intelligence".
We are going to be stuck in a few years of delusion, but there will be some entertainment as a side effect. AI executives and researchers are going to delve even deeper into AI psychosis thinking that the equivalent of psychedelic trip-looking school buses at 4000x zoom of an orange cat's ass, except in the form of text, is actually something intelligent and we just don't have the capacity to understand the "super intelligence's thoughts", all the while pouring another few trillions.
I agree with this article and the idea of RSI has never made sense to me.
Intelligence requires being able to solve problems in the real world; things like code and math are just useful proxies.
The real world cannot give instantaneous feedback like a compiler or proof-checker can. Suppose a model wanted to develop a cure to cancer. Wouldn’t it take years to synthesize and go through clinical trials for every iteration, creating a massive bottleneck on how quickly it can improve?
And that’s something that can eventually be verified. What about things that are difficult or impossible to systematically verify (knowing how/where to look for new ideas, or even just “common sense”)? How would a self-improving model even know it’s going wrong?
The evidence provided in the article is also compelling. If even frontier models only have 80% success on short AI research tasks, what happens if the model doesn’t realize its mistake and builds off of the work it fucked up? Wouldn’t the error rate compound, making even small error rates hugely problematic? (And these error rates OpenAI are showing are not small.)
It seems to be a given at this point that any clearly defined and quickly verifiable task, AI can do. But “make yourself smarter” is not such a task, and I don’t see a world where letting agents loop on that forever would lead to an explosion (other than in cost).
I’d love to hear what others think about this, since I feel like I must be missing something if all the top researchers seem to strongly believe in RSI as a possibility.
I like Noah's blog in general (this is a guest post), but I think that trying to precisely time and phase a new technological development you are watching from within is a fool's errand.
Autonomous recursive self-improvement needs to happen. It doesn't have to happen tomorrow, or in 10 years even, but it has to happen sometime in the near future, or we're kinda screwed.
Why say this? Well, of the plausible futures from here, the worst one is where AI plateus just good enough to make us all useless (or just destroy the pipeline of a critical mass of capable humans for assisted RSI), but at a point dumb enough that we don't get human-superior AI. This might happen or not, but there's no way to predict when something that doesn't exist will, so it's one of those horrors that can keep you up at night. Some of those skeptical about RSI have likened it to the Great Filter.
That seems like an OK future to me? The AI-addled can have their fun, the rest of us can eventually step in to pick up the pieces. I know it can seem bleak at times but the knowledge hasn't evaporated, it's just that (IMO) the less-AI-enthusiastic are stepping back a bit to let the energy spend itself. And "LLMs can't become superintelligence" doesn't imply "self-improving machine intelligence is impossible", for that matter.
The probly is believing our problems are just calculus.
They are not. Our problems are akin to cellular automata, and just the same. We can project a few steps forward but we cannot say this configuration will evolve to that configuration.
Eugenics is similar; the belief that we can measure physical chatacteristics and selectbased on the same for some desirable future configuration is just a self delusion.
Orboids in a swarm the whole does not direct the sum.
These are all basics that humans struggle against. AI will just head in adirection, and itll be as meaningless as an automata swarm. Youll recognize payterns but they wont make you sure of the next step.
Great analysis. Clearly shows "advancement" is not exponential but logarithmic. Hence, "We need to slow down!!" as a cover for stagnating improvements.
Pet theory: 70% of what we're busy with brings no value outside of keeping people busy/employed, which means automating it and making it 100x faster also brings no value in the end, no matter how much "intelligence" you can deploy
This!
An economist might counter that any such business would loose market share. But, in actuality, a lot busineses are not competing for survival or even for market share but have it due to entrenchment. examples include: ISPs, healthcare, etc
On the one hand this leads to the mediocre world we live in but on the other hand, thank goodness that these businesses exist. Imagine if every company were run like Tesla/SpaceX. You thought the horrendous birth rate in China was bad...it would be nothing compared to this theoretical world everybody is always in dog eat dog jungle survival mode.
I was reminded about how delusional people are when I was hanging out with some MAGA supporters in the south this weekend. These people adore Elon Musk and what he's done and the way that he runs his companies. They get this mentality because they work in government where their jobs are very cushy, but there is a lot of incompetence and bureaucracy going on, and they hate it. I honestly believe they wouldn't enjoy living in a world where Elon Musk mentality is running everything, including their jobs. That delusion makes them think that a world where Elon Musk is running everyone else's job is better for them.
I'd argue that of the people that Elon hires, many of them probably grew up in an environment where they had the support, love, and care that they need to become great because their predecessors worked in occupation that are not competing for survival. Else you end up with situations like China where the youths just don't have the ability (due to time, money and energy) to put down roots and build up the next generation. Their predecessors planted the seeds and nurtured them to grow value that is then extracted by Elon working the people into the ground and tossing them aside when they are no longer useful.
You can have cutthroat competition between companies and good working conditions, in fact more competition raises demand for labor so workers get more bargaining power. If employers are in survival mode they hold on to experienced workers at all costs and desperately try to attract new ones.
The problem in China comes from a concentrating economy and corrupt government that barely enforces labor laws. Demand for labor is not low but the number of employers is, so if you want better conditions and get rejected or fired you have nowhere else to go. America is on the same path.
>If employers are in survival mode they hold on to experienced workers at all costs and desperately try to attract new ones.
Can you give some examples? A company in survival mode can just as easily freeze hiring, cut benefits, and squeeze the people it already has. Needing experienced workers doesn’t mean it can afford to keep them.
Spirit is the example that bothers me. I’m biased because I loved flying with them, but former employees often describe good pay and working conditions alongside those cheap fares. How do you sustain that when costs rise and there’s barely any cushion for something going wrong? The big airlines have their credit card and loyalty businesses to lean on. What happens to a company trying to compete mainly on affordable flights?
My concern is that treating workers well costs money, and customers generally choose the cheaper or better product. What keeps a company that cuts corners from undercutting one that takes care of its employees?
I asked something similar at a HOPE talk recently about a worker owned software consultancy. Their answer was basically that everyone shares the pain when business gets rough. That sounds reasonable, and I want that model to work. I’m just not sure how well it scales.
>Demand for labor is not low but the number of employers is.
I also don’t think “too few employers” explains China very well. Look at the brutal competition among Chinese car companies, or in coffee, restaurants, and consumer goods. There can be plenty of competing businesses without workers having much leverage, especially when applicants outnumber the jobs available.
Weak labor law enforcement absolutely matters. But I don’t see why fierce competition and bad working conditions couldn’t reinforce each other. “We have to survive” can become the excuse for demanding more from everyone.
I want companies to improve their products while paying people well and treating them decently. I’m questioning whether competition alone gets us there. What makes employers compete for workers by improving conditions, instead of competing for customers by squeezing workers harder?
[dead]
If you look at computer chess it improved steadily, there was a bit of a fuss when it beat Kasparov, then it kept going. People still play each other at chess much as before.
I figure it'll be like that with general intelligence - bit of as fuss as AI overtakes but humans will keep on humaning, making things, competing for status. Maybe like Jeeves and Wooster with us as Wooster and the AI as Jeeves. Most of their activities didn't bring much value apart from keeping people busy. AI will probably keep improving a few IQ points a year, not exploding but gradually getting pretty good.
For me the more interesting sci-fi is the merge / immortality stuff, curious how that goes.
Some sectors are less productive, but the reality is more mundane: only about 30% of the global workforce are knowledge workers. Adopting AI in real workflows takes time, and that time saved might not be used productively. Also solving Millennium problems is an amazing feat of engineering and mathematics, but doesn't change people's lives that much.
Like advertising[1].
1. "Half the money I spend on advertising is wasted; the trouble is I don't know which half." - John Wanamaker
Many are actively harmful (fex. bringing about the climate catastrophe).
There is a name for this phenomena. It is called bullshit jobs[1]. The author David Graeber who coined the term argued that over half of jobs were bullshit jobs, so your 70% estimate is not far from what some actual scholars in the field also estimate.
1: https://en.wikipedia.org/wiki/Bullshit_Jobs
I don't want to discount the amazing progress we've made with technology to improve people's lives.
There's the meme that the smartest people in the world are figuring out how to show ads. It's no different this time around. I understand that not everyone working on AI is building an ad network - but ultimately it feels like that's the end result.
It didn't feel quite this way 15 years ago. Back then, it felt much more tangible how technology was improving our lives. It is sort of tangible now too - I am doing things with AI that I otherwise couldn't. But somehow if feels like progress for the sake of progress.
And to be honest, it doesn't feel like the progress is for me or the people I care about.
> ... it doesn't feel like the progress is for me or the people I care about.
AI is doing very little to improve the human condition; all it seems to be good at is optimizing productivity and generating profits somewhere maybe. That is not an AI-generated problem; using profits as an all-things compass is a disease we have been cooking for ourselves for a long time. But I'm sort of hoping AI will make the problem so bad that we will be forced to fix it, or perish.
That’s one of the main points in Karp’s book. All the smart people are happy to spend all day pushing gooning content on children or triggering alcoholics to relapse, but ask them to tackle crime or defend the country and they’ll tell you it’s immoral.
Nobody tries to solve any actual problems anymore, they just produce meaningless consumer slop. And when people like Musk try to invent reusable rockets or electric cars, everyone says they’re stupid. And then they do it, and everyone says it was always going to happen anyway.
When is the last time Google or meta or whoever actually solved a problem? They’ll spend 500b in a year, enough to reforest the entire planet, and what do they actually give us? I couldn’t name a single thing they’ve done in the last ten years which actually produces value, but they’ve spent trillions.
[dead]
https://washingtoncrossingadvisors.com/wp-content/uploads/20...
It has exploded, the explosion just isn't evenly distributed. And just as it was with the internet, most people and even most industries are not actually using it yet. Many that are are using it in name only.
People can star github repos and signup their employees for a new subscription quickly, real adoption takes far longer.
I am a smooth brained operator. very much the definition of vibecoding.
Last night I was able to give claude a task and they increased the classification accuracy from
81.3/89.9/92.1 to 90.9/99.3/99.6
In a little over 2 hours. In a task that up until opus 5.5 I was doing by hand/manually. and the day before was able to get the first results.
The intelligence explosion may just be user error.
Not where, when.
Asking where is the intelligence explosion suffers the same issue as the Gorman Paradox[1] - the question proceeded the explosion by six months. Timing explosions is like timing the market - infinitely harder than predicting eventual existence.
1. https://codemanship.wordpress.com/2025/12/14/the-gorman-para...
Even though I started using LLMs as soon as they became available, I was until recently very cautious around using them autonomously. I was of the opinion that I needed to provide heavy supervision around coding tasks. Granted, that was probably justified based on the model capabilities at the time.
However, lately, I have had several experiences that have completely changed my mind - the here is a high level goal (involving gathering production logs, setting up an eval, running tests, iterating, including diagnosis and coming up with new ideas, etc. and don’t bother me until it’s done. The experience has been incredible.
This has been possible because the agent has a way to iterate and hillclimb against a metric it can measure. Relatively easy in the world of software.
I’m now convinced that we will unlock the same gains once make other domains similarly “iterable”.
Eg Agent comes up with 100 new proteins, runs in lab autonomously, gathers results, iterates. One month (or whatever) later, you have your new protein designed.
I think we're on the cusp of an intelligence explosion, but most jobs have no real purpose besides giving us sometime to do and a paycheck. Automating those jobs and making them 10x faster won't really help anyone. Read Bullshit Jobs by Graeber.
FWIW, why I'm rooting hard for AGI: -In the best case, AGI will lead to UBI, so unemployment for all becomes the default scenario. Sounds scary but unemployment is better than many soul-sucking occupations, e.g. cold-calling. -If AGI does result in a productivity boom due to automation, we're in for a deflationary spiral, so we don't need a huge amount of money for a decent standard of living.
It does lead to the question of how we'll find meaning in our lives, but I think we'll adapt in a few years.
If you’re a doctor and you have to work five days a week, why would you keep working if 99% of the population just sits at home wanking all day?
If you’re the robot owner, why wouldn’t you just cull the population and build Jurassic park?
There’s just no way it’s feasible to pay the entire population to sit around.
Navier stokes isn't enough, let's wait for quantum and general relativity solution but it will also not be enough. Maybe a Dyson sphere?
As the entire universe gets converted into one gigantic paperclip, the last message written by humanity will probably be an article expressing skepticism whether AI progress is real or it's all just a marketing ploy.
Just like when you upscale a picture 100 times by 200%, and then you zoom in and start seeing all these artifacts, so is the outcome of "recursive intelligence".
We are going to be stuck in a few years of delusion, but there will be some entertainment as a side effect. AI executives and researchers are going to delve even deeper into AI psychosis thinking that the equivalent of psychedelic trip-looking school buses at 4000x zoom of an orange cat's ass, except in the form of text, is actually something intelligent and we just don't have the capacity to understand the "super intelligence's thoughts", all the while pouring another few trillions.
I agree with this article and the idea of RSI has never made sense to me.
Intelligence requires being able to solve problems in the real world; things like code and math are just useful proxies.
The real world cannot give instantaneous feedback like a compiler or proof-checker can. Suppose a model wanted to develop a cure to cancer. Wouldn’t it take years to synthesize and go through clinical trials for every iteration, creating a massive bottleneck on how quickly it can improve?
And that’s something that can eventually be verified. What about things that are difficult or impossible to systematically verify (knowing how/where to look for new ideas, or even just “common sense”)? How would a self-improving model even know it’s going wrong?
The evidence provided in the article is also compelling. If even frontier models only have 80% success on short AI research tasks, what happens if the model doesn’t realize its mistake and builds off of the work it fucked up? Wouldn’t the error rate compound, making even small error rates hugely problematic? (And these error rates OpenAI are showing are not small.)
It seems to be a given at this point that any clearly defined and quickly verifiable task, AI can do. But “make yourself smarter” is not such a task, and I don’t see a world where letting agents loop on that forever would lead to an explosion (other than in cost).
I’d love to hear what others think about this, since I feel like I must be missing something if all the top researchers seem to strongly believe in RSI as a possibility.
I like Noah's blog in general (this is a guest post), but I think that trying to precisely time and phase a new technological development you are watching from within is a fool's errand.
Autonomous recursive self-improvement needs to happen. It doesn't have to happen tomorrow, or in 10 years even, but it has to happen sometime in the near future, or we're kinda screwed.
Why say this? Well, of the plausible futures from here, the worst one is where AI plateus just good enough to make us all useless (or just destroy the pipeline of a critical mass of capable humans for assisted RSI), but at a point dumb enough that we don't get human-superior AI. This might happen or not, but there's no way to predict when something that doesn't exist will, so it's one of those horrors that can keep you up at night. Some of those skeptical about RSI have likened it to the Great Filter.
That seems like an OK future to me? The AI-addled can have their fun, the rest of us can eventually step in to pick up the pieces. I know it can seem bleak at times but the knowledge hasn't evaporated, it's just that (IMO) the less-AI-enthusiastic are stepping back a bit to let the energy spend itself. And "LLMs can't become superintelligence" doesn't imply "self-improving machine intelligence is impossible", for that matter.
What happens if the momentum of us AI-addled outlives the tribal knowledge on the sidelines?
I genuinely fear your scenario -- "AI was good enough for long enough to let the artisans with the knowledge die off" -- now it's idiocracy. :)
Although is this different than people who know how to build and operate a steam engine? I am not certain.
The probly is believing our problems are just calculus.
They are not. Our problems are akin to cellular automata, and just the same. We can project a few steps forward but we cannot say this configuration will evolve to that configuration.
Eugenics is similar; the belief that we can measure physical chatacteristics and selectbased on the same for some desirable future configuration is just a self delusion.
Orboids in a swarm the whole does not direct the sum.
These are all basics that humans struggle against. AI will just head in adirection, and itll be as meaningless as an automata swarm. Youll recognize payterns but they wont make you sure of the next step.
Great analysis. Clearly shows "advancement" is not exponential but logarithmic. Hence, "We need to slow down!!" as a cover for stagnating improvements.
it seems that the big leap has already ended. Now its more cutting costs and improving speed. But maybe it is better this way...
dupe of https://news.ycombinator.com/item?id=49883815
RSI is way closer to reality than "nanotechnological telepathy distributed as a party drug"