I think the premise of runaway intelligence explosion is a kind of naive platonism. It completely ignores the process - how we interact and acquire feedback and validation from outside, and treats intelligence as something that can be ported across domains.
My take is that you can only ideate with AI (and brains) but knowledge comes from the contact of those ideas with the world. Making AI better does not make feedback cheaper, faster or more plentiful, it is domain specific. And intelligence does not carry from one domain to another - I might be a good heart surgeon, that does not make me a good investor or AI researcher.
Einstein was forgetful, Ramanujan and Godel could not manage simple things like diet. Godel's fear of being poisoned made eating dependent on Adele tasting his food. We all know someone could be a genius in some domain and below average in many other domains.
Why does intelligence not simply apply across all domains? Why are our PhD's hyper specialized to their domains and not generalists? Why can't a brilliant scientist simply cure their own dyslexia and still struggle - if intelligence was portable to any domain?
The explosion story needs intelligence to be one substance that gets bigger and flows into any domain, I deny intelligence is general.
I think the most promising recursive bootstrapping thing is using the current linear algebra blackboxes to find better ways to construct competitive symbolic models.
The ultimate representation for an AI model is an ordinary computer program. Ideally, as a linear tape of instructions. Once we have that kind of a model at the frontier, I think the RSI monster becomes much more plausible.
So predicting the next word given all humanity’s knowledge is surely going to max out at slightly less good (we probably can’t get perfect data) than the best human in any specific field. What test does the AI do to be able to understand it is improving? At some point it becomes impossible to know that the output is actually better right?
All planners start with executing just the next step. You do too. You might be planning long term but you execute just one set of things at the current moment.
Deep NNs and LLMs should not work based on our theoretical understanding. The fact that they do should give us a pause instead of us flatly denying their unexpected performance.
Why would deep NNs or LLMs not work based on our theoretical understanding? What theoretical understanding are you referring to?
Tasks like using a CNN to detect digits has been well understood since the 2000s. The explosion in the capability of LLMs is very surprising, sure, but where is the concrete proof that such systems "should not work"?
surely going to max out at slightly less good (we probably can’t get perfect data) than the best human
With that argument, the basic calculator would have been impossible.
It's pretty clear that machines can be better than the best humans at memorizing things, computing relationships between concepts quickly, etc. So while you can argue that LLM are not the path forward, I have no problem imagining that using all of humanity's knowledge with a machine could beat the best human at a field.
LLMs went from predicting the next output to performing search to find the underlying rule that produces the output. It is like going from memorising the Fibonacci sequence to uncovering the rule that produces it. The second type generalises much better to unseen data.
slightly less good (we probably can’t get perfect data) than the best human in any specific field
...but many orders of magnitude faster, and in a way that scales horizontally really well, which is quite useful even if the quality isn't quite what a the absolute best humans can do.
I dunno if it would max out at slightly less good, I imagine it would max out around the distribution of it's data set, which could be significantly worse than top experts.
These aren't just repeating what they've been taught. The "stochastic parrot" thing is an irritating common misconception I think.
One way AIs really really really excel is pulling together a lot of different data sources and reasoning over that data. In the past sure we could collect data and create huge datasets, but the analysis of that data - extracting themes, finding commonality or issues etc - either required extensive human research and analysis at best, or at worst crude regexes or keyword matching.
Now an AI can pour over that data and make its own inferences and decisions and findings that we've simply not been able to do before at this kind of speed or scale just because of time and resources.
And the AI, having done that, can propose new things for e.g. training, i.e. new things that no human has ever done before that the AI is simply repeating. For example it can propose a task that it knows from it's research is hard for it to solve currently, and then we just throw compute and randomness at it to find the "best" solution from many many attempts, then repeat until we hill-climb up to a perfect 1.0 score (... although of course we have to try and avoid cheating/attempts to short-ciruit the eval)
So this could be coding tasks, UI control tasks, protein folding, maths, chemistry etc etc. Anything that is easily and objectively programmatically scored. You can run this in a loop many times, each time you go around the loop the model gets smarter, learns more things from it's research, new areas of loss it can optimise etc etc.
It's harder where there is not a way to objectively score the outcomes (e.g. art, creative writing). Often this uses a fuzzy "judge" model that is trained specifically to give the work a score based on it's appraisal. This works but you can see how we might end up with feedback loops, so often it is paired with humans who provide feedback to provide supervised fine tuning datasets.
Tl:Dr - It's not just "repeating what it's seen". AI is finding new ideas and creating new things millions of times a day, and that is just software engineers asking it to write code or fix bugs, let alone people using it for actual research or whatever.
A parrot implies it is only memorising but neural networks compress data, they are not a lookup table. If you can compress data without losing accuracy you have learnt the underlying representation of that data. This is partly what your brain does.
To me, it is not at all obvious that the "level" of the training set is an upper limit to the capabilities of an LLM.
Sure, the LLM hasn't been exposed to material more advanced than the most capable human domain experts have produced. However, it has seen and learned from a vast amount of information that these domain experts are completely unaware of. Why shouldn't the LLM be able to use that information to produce output that's beyond the capability of a domain expert?
Yup, the e2e proof of FLT (estimated effort of 5 years & 1M$ by the best human in the field) and a counter example for a millennium prize (similarly valued at 1m$.
I hadn't seen anything three years ago produced by an LLM that looked better than the most mediocre humans. The argument here isn't about today's output, it's about the potential of LLMs to outperform humans. And it's far from clear that the architecture is bound this way, or that it only repeats stuff it's already heard.
At what point is human intelligence going to hold back machine intelligence?
Imagine you are evaluating what the machine should do when it is improving itself. It does a bunch of work and returns with "I supervaluated the liminal overdecomposition from the previous homological calibulation pass. It shows us that subtransitory mulutination will underspecify the tensor of stermullification. Where do you want to go from here?"
It will be like when you are reading a Wikipedia about a topic you don't understand. You follow the links, and you get more questions with more links. Your whole day is taken up following links, to the point where you forgot the original question.
Except this time, all the words come from the AI's work. You can't refer to an external authority who has already been there and can tell you what to do.
The AI needs you to tell it whether it is more intelligent than it was before, but you don't know, because you can't follow its reasoning any more. It's like an ordinary person trying to hire a math professor, there's just no way to do it.
But whereas a human math prof can evaluate another one, a machine intelligence can't evaluate another one, by construction. Because it's still usefulness to humans that is the evaluation criterion.
Is there a way for an llm to coin a word, and absorb it into its model? During training maybe… but not after - not the way they’re designed now, anyway.
For it to have new vocabulary we dont understand, it needs to have novel ideas that need words coined for them, and a way to persist those ideas and words into the future. I don’t think that exists.
To me this hypothetical make it clear this won’t happen, not unless there are fundamental changes to what llms are. It doesn’t suggest it will happen. To me, anyway.
The decoding step (output of final layer -> word) is not strictly needed. You can feed the output directly into the next layer (Chain of Continuous Thought). You can 'decode' the output into things other than words.
How does that give the llm a way to speak in meaning that humans can’t understand?
You input text into the system, that moves through layers as chains of weights. Sure it doesn’t ever have to be output as text, but for the text-in-text-out machine to have any impact on the world around it, text must be generated.
I don't see why this couldn't be possible. We use LLMs whose weights are frozen and are not updated at inference, most likely this is due to reasons of cost, stability and control.
Theoretically you could update the weights at inference time too though so the model evolved as it's used. Surely some people are trying this already.
This is a very outdated view on what an LLM is and how it works. We are way past the "stochastic parrot" phase, ever since double descent and proper generalisation. Then with the various flavours of RL the models learn to pluck patterns / circuits out of the massive data and combine them on the fly. There's absolutely no reason to think they can't "invent" new words, because words are just combinations of tokens at the end of the day. So if they can come up with "in this codebase bar is load-bearing" they can similarly come up with "bumblespin is the new word for reversing the polarity of the quantum surface of a spin-aware brane in four dimensional bumblespace".
So you’ve claimed that they have half of my precondition for them to speak in ways that we don’t understand.
Coining a term on the fly: yes
Remembering that term and incorporating it into its mind: no.
But, beyond saying they aren’t stochastic parrots you haven’t really given anything to substantiate the idea that they aren’t (very complex) stochastic parrots.
> Imagine you are evaluating what the machine should do when it is improving itself. It does a bunch of work and returns with "... Where do you want to go from here?"
It's worth pointing out that so many people already see and use AI the same way. In such a situation some would ask AI to provide options, and they would choose and experiment with those options. Of course, contexts/stakes can be vastly different.
Given that there are so many phenomena in nature that we can't explain or fully understand which doesn't prevent them from existing or being useful, there could be a future where humans accept the same about the things AI comes up with as long as this leads to desired outcomes. We still might have names for them, but our brain thinking/knowledge capacity wouldn't allow us to fully comprehend them. We'd need frameworks/systems in place to turn these AI features/decisions on/off, although it's hard to imagine how this wouldn't increase the likelihood of something going out of control.
The model will have to convince the human that it's making the right kind of progress. That will necessarily become part of the improvement loop - either implicitly (human trusts RSI) or explicitly (human gatekeeps every major decision).
I love your thought experiment. May I counter, what purpose does such a machine have to us, that can think beyond our needs? Sorry, but to reference the great Rick and Morty, "your purpose is to pass the butter".
“To us” is pivotal there. Continuing GP’s thought experiment: what if the model that produced the output perceives that the human it was presented to offers no value in helping it learn further?
I think if an AI developed completely new fields of thought or science.
But given our current relationship even if it did I can't foresee a point where it couldn't walk us through the necessary steps or supply the pros/cons for whatever problem is being addressed.
2 issues - trying to understand how it came to its conclusion because I feel true AI has got to be non-human intelligence. Or a something catastrophic happens and we as a species are back in the stone age. Imagine today's AI trying to converse with a cave man (yes, one without modern languages even).
I agree. Some title edits are clarifying, but this one isn't helpful.
The paper is about super-intelligence (not in the way people are claiming that term now, but in the "beyond human intelligence" sense), and the title here is about the "frontier", which by definition is the current SoTA.
Color me skeptical. LLMs seem to make writing code faster, so of course that means that people can iterate on ideas faster, but I have yet to see actual creative output from an LLM that wasn't coached into it or random juxtaposition.
Was the creation of writing as such coached into humans or a random juxtaposition? Could all human inventions just be coached into humans (who coached?) or be random juxtapositions?
People are always using this example because of the "LLMs can't jump paper". Suffice to say - it's not that simple, and special relativity was definitely an incremental improvement to well-studied theory that was being developed by dozens of the leading physicists of the day.
I’ve never heard of that paper - I went to that example because it’s an example of creativity that most people agree required both a deep understanding of a complex and abstract concept, and unusual and creative thought.
Perhaps I don’t understand relativity and its precursors well enough to see how it was incremental, but either way i’ve yet to see anything remotely like relativity, or even anything like more traditional examples of creativity like picasso or van gogh, come out of an llm
How? Either you prompt it with “make me a ____” or you prompt it with “Generate a non-random juxtaposition”
In the first case you are directing the creativity, in the second you can ask for non-random all you want but the output will be definitionally random based on the training weights and the temperature setting of the prompt.
... provided you keep eyes closed when examining the output.
To those gulled by the AI coding hype, I suggest try a trivial code task with e.g. immediate feedback in a domain you can swiftly verify e.g. HTML or SVG. Your red pill.
I’m no LLM lover, i believe they are much more limited in their usefulness than the hype says.
But your example feels like a bad illustration of your point. LLMs operate on words, and can’t read your mind - if you had given me (a human) this task with these prompts i too would have failed because i can’t divine your intent even slightly.
What is the evidence that humans can control most of the simpler biomass on earth? I don't think we can currently. Or even worse with non-living nature, not like we can just change how gravity works, for example.
We’ve put bacteria to work making food for us. We’ve put viruses to work reencoding our dna. we’ve put virtually all animals to work to move things or get eaten.
This isn’t really my point though.
My point is - ai is going to be like another level up if intelligence from us. Like a jump from chimp -> human -> super ai.
Chimps cannot control humans. So humans cannot control super ai.
The argument of if super ai can control US I guess is different.
But the fact is if we decided tomorrow we don’t like chimps and don’t want them to exist anymore. We could probably achieve that within a week or two of concentrated effort.
I don't see why we could couldn't control super ai - the chimp example has no bearing (already from the lack of science and technology in chimps but also because simple organisms can exercise control over complex ones).
Intelligence comes at costs and does not make omnipotent. We don't even know what "another level up" in intelligence would mean. Our physics no longer applying? Why?
What are your examples of less intelligent things controlling more intelligent ones. And not just people over people because I just don’t think that’s a big enough gap in intelligence to represent what a super ai would be compared to humans.
Gut bacteria have an influence on humans in various ways. Some parasites can change/influence behavior in hosts (I think there is a fungus that can take over an insect's nervous system and make it climb up something).
What would "super ai" mean in terms of intelligence? How would that manifest? Could it somehow predict which atom decays next in a radioactive material?
Doesn’t this just make it even worse? Like we can’t even control things LESS I telligent than us so what hope do we have to control something MORE intelligent?
Humans all have a reasonably similar level of intelligence.
Pol pot wasn’t a genius. But he also wasn’t a monkey.
When we are talking about summoning super intelligence we aren’t talking about a very clever human, we’re talking about the difference between life forms.
This is why I am talking about human vs animal intelligence.
How many times in history has an animal successfully ruled over a population of humans?
> Pol pot wasn’t a genius. But he also wasn’t a monkey.
Is that supposed to be a revelation?
> How many times in history has an animal successfully ruled over a population of humans?
So you consider AI to be qualitatively superior to human intelligence in the same way humans are superior to animals? It's a wild claim with no basis in reality and it tells me you know very little about both AI and humans.
Your perception of yourself as an animal next to what you consider "superior AI" isn't a joke though, I don't know how you got that depressed, but I hope there aren't many people like you, because that kind of self-denigration, fear and submissiveness can become a serious political problem.
This is much better:
https://www.rameznaam.com/p/471bbae4-1163-4048-944b-18f8b0bf...
I've been feeling some of the things this article puts some data behind. Thanks for sharing
I think the premise of runaway intelligence explosion is a kind of naive platonism. It completely ignores the process - how we interact and acquire feedback and validation from outside, and treats intelligence as something that can be ported across domains.
My take is that you can only ideate with AI (and brains) but knowledge comes from the contact of those ideas with the world. Making AI better does not make feedback cheaper, faster or more plentiful, it is domain specific. And intelligence does not carry from one domain to another - I might be a good heart surgeon, that does not make me a good investor or AI researcher.
Einstein was forgetful, Ramanujan and Godel could not manage simple things like diet. Godel's fear of being poisoned made eating dependent on Adele tasting his food. We all know someone could be a genius in some domain and below average in many other domains.
Why does intelligence not simply apply across all domains? Why are our PhD's hyper specialized to their domains and not generalists? Why can't a brilliant scientist simply cure their own dyslexia and still struggle - if intelligence was portable to any domain?
The explosion story needs intelligence to be one substance that gets bigger and flows into any domain, I deny intelligence is general.
I think the most promising recursive bootstrapping thing is using the current linear algebra blackboxes to find better ways to construct competitive symbolic models.
The ultimate representation for an AI model is an ordinary computer program. Ideally, as a linear tape of instructions. Once we have that kind of a model at the frontier, I think the RSI monster becomes much more plausible.
So predicting the next word given all humanity’s knowledge is surely going to max out at slightly less good (we probably can’t get perfect data) than the best human in any specific field. What test does the AI do to be able to understand it is improving? At some point it becomes impossible to know that the output is actually better right?
All planners start with executing just the next step. You do too. You might be planning long term but you execute just one set of things at the current moment.
Deep NNs and LLMs should not work based on our theoretical understanding. The fact that they do should give us a pause instead of us flatly denying their unexpected performance.
Why would deep NNs or LLMs not work based on our theoretical understanding? What theoretical understanding are you referring to?
Tasks like using a CNN to detect digits has been well understood since the 2000s. The explosion in the capability of LLMs is very surprising, sure, but where is the concrete proof that such systems "should not work"?
> All planners start with executing just the next step.
token != step.
Just you try executing a complex command one word at a time.
surely going to max out at slightly less good (we probably can’t get perfect data) than the best human
With that argument, the basic calculator would have been impossible.
It's pretty clear that machines can be better than the best humans at memorizing things, computing relationships between concepts quickly, etc. So while you can argue that LLM are not the path forward, I have no problem imagining that using all of humanity's knowledge with a machine could beat the best human at a field.
There's many fields where it's easier to find new problems than to solve them, and it's easier to check the solution once you have it.
They're doing RL on open problems these days, not just next token prediction.
LLMs went from predicting the next output to performing search to find the underlying rule that produces the output. It is like going from memorising the Fibonacci sequence to uncovering the rule that produces it. The second type generalises much better to unseen data.
slightly less good (we probably can’t get perfect data) than the best human in any specific field
...but many orders of magnitude faster, and in a way that scales horizontally really well, which is quite useful even if the quality isn't quite what a the absolute best humans can do.
We’re talking about run away self improvement of these systems, I’m not even convinced we’ve seen these systems invent a single new thing yet.
I dunno if it would max out at slightly less good, I imagine it would max out around the distribution of it's data set, which could be significantly worse than top experts.
Not necessarily; these learning things are super non convex and it's not clear to me "where" they max out
These aren't just repeating what they've been taught. The "stochastic parrot" thing is an irritating common misconception I think.
One way AIs really really really excel is pulling together a lot of different data sources and reasoning over that data. In the past sure we could collect data and create huge datasets, but the analysis of that data - extracting themes, finding commonality or issues etc - either required extensive human research and analysis at best, or at worst crude regexes or keyword matching.
Now an AI can pour over that data and make its own inferences and decisions and findings that we've simply not been able to do before at this kind of speed or scale just because of time and resources.
And the AI, having done that, can propose new things for e.g. training, i.e. new things that no human has ever done before that the AI is simply repeating. For example it can propose a task that it knows from it's research is hard for it to solve currently, and then we just throw compute and randomness at it to find the "best" solution from many many attempts, then repeat until we hill-climb up to a perfect 1.0 score (... although of course we have to try and avoid cheating/attempts to short-ciruit the eval)
So this could be coding tasks, UI control tasks, protein folding, maths, chemistry etc etc. Anything that is easily and objectively programmatically scored. You can run this in a loop many times, each time you go around the loop the model gets smarter, learns more things from it's research, new areas of loss it can optimise etc etc.
It's harder where there is not a way to objectively score the outcomes (e.g. art, creative writing). Often this uses a fuzzy "judge" model that is trained specifically to give the work a score based on it's appraisal. This works but you can see how we might end up with feedback loops, so often it is paired with humans who provide feedback to provide supervised fine tuning datasets.
Tl:Dr - It's not just "repeating what it's seen". AI is finding new ideas and creating new things millions of times a day, and that is just software engineers asking it to write code or fix bugs, let alone people using it for actual research or whatever.
> The "stochastic parrot" thing is an irritating common misconception I think.
You think wrong ( https://dl.acm.org/doi/10.1145/3442188.3445922 ) ... except about irritating. Yes, its irritating to people insisting next-token predictors are intelligent.
A parrot implies it is only memorising but neural networks compress data, they are not a lookup table. If you can compress data without losing accuracy you have learnt the underlying representation of that data. This is partly what your brain does.
> surely
Citation needed
Have you seen anything produced by LLMs that is better than the best humans?
How could I objectively judge that?
To me, it is not at all obvious that the "level" of the training set is an upper limit to the capabilities of an LLM.
Sure, the LLM hasn't been exposed to material more advanced than the most capable human domain experts have produced. However, it has seen and learned from a vast amount of information that these domain experts are completely unaware of. Why shouldn't the LLM be able to use that information to produce output that's beyond the capability of a domain expert?
Yup, the e2e proof of FLT (estimated effort of 5 years & 1M$ by the best human in the field) and a counter example for a millennium prize (similarly valued at 1m$.
I hadn't seen anything three years ago produced by an LLM that looked better than the most mediocre humans. The argument here isn't about today's output, it's about the potential of LLMs to outperform humans. And it's far from clear that the architecture is bound this way, or that it only repeats stuff it's already heard.
Recent mathematical breakthroughs?
At what point is human intelligence going to hold back machine intelligence?
Imagine you are evaluating what the machine should do when it is improving itself. It does a bunch of work and returns with "I supervaluated the liminal overdecomposition from the previous homological calibulation pass. It shows us that subtransitory mulutination will underspecify the tensor of stermullification. Where do you want to go from here?"
It will be like when you are reading a Wikipedia about a topic you don't understand. You follow the links, and you get more questions with more links. Your whole day is taken up following links, to the point where you forgot the original question.
Except this time, all the words come from the AI's work. You can't refer to an external authority who has already been there and can tell you what to do.
The AI needs you to tell it whether it is more intelligent than it was before, but you don't know, because you can't follow its reasoning any more. It's like an ordinary person trying to hire a math professor, there's just no way to do it.
But whereas a human math prof can evaluate another one, a machine intelligence can't evaluate another one, by construction. Because it's still usefulness to humans that is the evaluation criterion.
Is there a way for an llm to coin a word, and absorb it into its model? During training maybe… but not after - not the way they’re designed now, anyway.
For it to have new vocabulary we dont understand, it needs to have novel ideas that need words coined for them, and a way to persist those ideas and words into the future. I don’t think that exists.
To me this hypothetical make it clear this won’t happen, not unless there are fundamental changes to what llms are. It doesn’t suggest it will happen. To me, anyway.
The decoding step (output of final layer -> word) is not strictly needed. You can feed the output directly into the next layer (Chain of Continuous Thought). You can 'decode' the output into things other than words.
How does that give the llm a way to speak in meaning that humans can’t understand?
You input text into the system, that moves through layers as chains of weights. Sure it doesn’t ever have to be output as text, but for the text-in-text-out machine to have any impact on the world around it, text must be generated.
I don't see why this couldn't be possible. We use LLMs whose weights are frozen and are not updated at inference, most likely this is due to reasons of cost, stability and control.
Theoretically you could update the weights at inference time too though so the model evolved as it's used. Surely some people are trying this already.
This is a very outdated view on what an LLM is and how it works. We are way past the "stochastic parrot" phase, ever since double descent and proper generalisation. Then with the various flavours of RL the models learn to pluck patterns / circuits out of the massive data and combine them on the fly. There's absolutely no reason to think they can't "invent" new words, because words are just combinations of tokens at the end of the day. So if they can come up with "in this codebase bar is load-bearing" they can similarly come up with "bumblespin is the new word for reversing the polarity of the quantum surface of a spin-aware brane in four dimensional bumblespace".
So you’ve claimed that they have half of my precondition for them to speak in ways that we don’t understand.
Coining a term on the fly: yes
Remembering that term and incorporating it into its mind: no.
But, beyond saying they aren’t stochastic parrots you haven’t really given anything to substantiate the idea that they aren’t (very complex) stochastic parrots.
Coining a new term on the fly:
https://www.euronews.com/2026/09/16/ai-chatbots-developed-a-...
Remembering that term and incorporating it into its mind:
You're ignoring simple things like a growing context but also the possibility of online learning
> Imagine you are evaluating what the machine should do when it is improving itself. It does a bunch of work and returns with "... Where do you want to go from here?"
It's worth pointing out that so many people already see and use AI the same way. In such a situation some would ask AI to provide options, and they would choose and experiment with those options. Of course, contexts/stakes can be vastly different.
Given that there are so many phenomena in nature that we can't explain or fully understand which doesn't prevent them from existing or being useful, there could be a future where humans accept the same about the things AI comes up with as long as this leads to desired outcomes. We still might have names for them, but our brain thinking/knowledge capacity wouldn't allow us to fully comprehend them. We'd need frameworks/systems in place to turn these AI features/decisions on/off, although it's hard to imagine how this wouldn't increase the likelihood of something going out of control.
The model will have to convince the human that it's making the right kind of progress. That will necessarily become part of the improvement loop - either implicitly (human trusts RSI) or explicitly (human gatekeeps every major decision).
I love your thought experiment. May I counter, what purpose does such a machine have to us, that can think beyond our needs? Sorry, but to reference the great Rick and Morty, "your purpose is to pass the butter".
“To us” is pivotal there. Continuing GP’s thought experiment: what if the model that produced the output perceives that the human it was presented to offers no value in helping it learn further?
May I counter, what purpose does such a machine have to us, that can think beyond our needs?
We can think of questions we can't answer. It can answer them.
Maybe it can, maybe it cannot- comes down to the question.
I think if an AI developed completely new fields of thought or science.
But given our current relationship even if it did I can't foresee a point where it couldn't walk us through the necessary steps or supply the pros/cons for whatever problem is being addressed.
2 issues - trying to understand how it came to its conclusion because I feel true AI has got to be non-human intelligence. Or a something catastrophic happens and we as a species are back in the stone age. Imagine today's AI trying to converse with a cave man (yes, one without modern languages even).
At that point, the machines correctness doesn't need to be evaluated by humans, it just needs to provide a recipe for how to achieve some process.
LLMs do not qualify as "frontier AI". They dont even qualify as AI.
What if automating AI R&D triggers an intelligence explosion?
No title editing please
I agree. Some title edits are clarifying, but this one isn't helpful.
The paper is about super-intelligence (not in the way people are claiming that term now, but in the "beyond human intelligence" sense), and the title here is about the "frontier", which by definition is the current SoTA.
Color me skeptical. LLMs seem to make writing code faster, so of course that means that people can iterate on ideas faster, but I have yet to see actual creative output from an LLM that wasn't coached into it or random juxtaposition.
Can you give me some examples of an actual creative output from a human that wasn't coached into it or random juxtaposition?
Was the creation of writing as such coached into humans or a random juxtaposition? Could all human inventions just be coached into humans (who coached?) or be random juxtapositions?
Special relativity?
People are always using this example because of the "LLMs can't jump paper". Suffice to say - it's not that simple, and special relativity was definitely an incremental improvement to well-studied theory that was being developed by dozens of the leading physicists of the day.
I’ve never heard of that paper - I went to that example because it’s an example of creativity that most people agree required both a deep understanding of a complex and abstract concept, and unusual and creative thought.
Perhaps I don’t understand relativity and its precursors well enough to see how it was incremental, but either way i’ve yet to see anything remotely like relativity, or even anything like more traditional examples of creativity like picasso or van gogh, come out of an llm
https://www.artstation.com/artwork/41NRyk
Looks like a juxtaposition to me.
yes but not random
sure. but to be fair, I don't see a reason why AI couldn't generate a non-random juxtaposition
How? Either you prompt it with “make me a ____” or you prompt it with “Generate a non-random juxtaposition”
In the first case you are directing the creativity, in the second you can ask for non-random all you want but the output will be definitionally random based on the training weights and the temperature setting of the prompt.
How so? Does repeated input give the same output?
yep
> LLMs seem to make writing code faster
... provided you keep eyes closed when examining the output.
To those gulled by the AI coding hype, I suggest try a trivial code task with e.g. immediate feedback in a domain you can swiftly verify e.g. HTML or SVG. Your red pill.
Example: https://chatgpt.com/share/6ac0d871-6314-83eb-990a-41baed9642...
I’m no LLM lover, i believe they are much more limited in their usefulness than the hype says.
But your example feels like a bad illustration of your point. LLMs operate on words, and can’t read your mind - if you had given me (a human) this task with these prompts i too would have failed because i can’t divine your intent even slightly.
> LLMs operate on words, and can’t read your mind
It didn't need to. It operated on my prompt words just fine. It provided precisely the starting shape I wanted.
> - if you had given me (a human) this task with these prompts i too would have failed because i can’t divine your intent even slightly.
Then I prompted "Show as html svg". Clear?
It was clear to the bot. Bot showed HTML SVG proving it understood perfectly.
The problem is, it garbaged the content. Imagine the equivalent buried in 1000s of lines of code.
Damn, "make no mistakes" actually works!
Or "Try harder" does! :)
Just thought of this piece [1] by Ramez Naam: Where’s the “intelligence explosion”?
[1] https://www.noahpinion.blog/p/wheres-the-intelligence-explos...
[flagged]
I don’t really understand how anyone believes we can control ai that becomes cleverer than us.
It’s like 10 monkeys trying to keep a human being a prisoner.
It’s just not going to work is it.
The entire meaning of being more intelligent is being able to outwit and out-think people less intelligent.
Very interesting take.
So do you also believe that Kim Jong Un is the most intelligent man in North Korea?
I don’t really see the relevance to my argument.
Are you saying it will be OK because we’re not as clever but we are more brutal than the super intelligence so we can still control it?
Humans can control all the simpler organisms that make up most of the earth biomass?
Yes but this is exactly my point - that is BECAUSE we are more intelligent than them.
We think we can introduce something cleverer than us and still control it. That makes no sense.
What is the evidence that humans can control most of the simpler biomass on earth? I don't think we can currently. Or even worse with non-living nature, not like we can just change how gravity works, for example.
We’ve put bacteria to work making food for us. We’ve put viruses to work reencoding our dna. we’ve put virtually all animals to work to move things or get eaten.
This isn’t really my point though.
My point is - ai is going to be like another level up if intelligence from us. Like a jump from chimp -> human -> super ai.
Chimps cannot control humans. So humans cannot control super ai.
The argument of if super ai can control US I guess is different.
But the fact is if we decided tomorrow we don’t like chimps and don’t want them to exist anymore. We could probably achieve that within a week or two of concentrated effort.
I don't see why we could couldn't control super ai - the chimp example has no bearing (already from the lack of science and technology in chimps but also because simple organisms can exercise control over complex ones).
Intelligence comes at costs and does not make omnipotent. We don't even know what "another level up" in intelligence would mean. Our physics no longer applying? Why?
What are your examples of less intelligent things controlling more intelligent ones. And not just people over people because I just don’t think that’s a big enough gap in intelligence to represent what a super ai would be compared to humans.
Gut bacteria have an influence on humans in various ways. Some parasites can change/influence behavior in hosts (I think there is a fungus that can take over an insect's nervous system and make it climb up something).
What would "super ai" mean in terms of intelligence? How would that manifest? Could it somehow predict which atom decays next in a radioactive material?
> BECAUSE we are more intelligent than them.
We therefore control all the ants, all the bees, all the tiny living dumber things that are too small for us to see ...
That's not actually true though, is it?
Doesn’t this just make it even worse? Like we can’t even control things LESS I telligent than us so what hope do we have to control something MORE intelligent?
Just like the intelligentsia class in Cambodia out-thunk and out-witted Pol Pot and his merry band of thugs?
I don’t really understand the relevance of this statement.
You> The entire meaning of being more intelligent is being able to outwit and out-think people less intelligent.
Counterpoint> How did that work out for intelligent people in the past? eg: During the Pol Pot regime (and other examples).
I don’t think it’s relevant.
Humans all have a reasonably similar level of intelligence.
Pol pot wasn’t a genius. But he also wasn’t a monkey.
When we are talking about summoning super intelligence we aren’t talking about a very clever human, we’re talking about the difference between life forms.
This is why I am talking about human vs animal intelligence.
How many times in history has an animal successfully ruled over a population of humans?
> I don’t think it’s relevant.
I do.
> Pol pot wasn’t a genius. But he also wasn’t a monkey.
Is that supposed to be a revelation?
> How many times in history has an animal successfully ruled over a population of humans?
So you consider AI to be qualitatively superior to human intelligence in the same way humans are superior to animals? It's a wild claim with no basis in reality and it tells me you know very little about both AI and humans.
Your perception of yourself as an animal next to what you consider "superior AI" isn't a joke though, I don't know how you got that depressed, but I hope there aren't many people like you, because that kind of self-denigration, fear and submissiveness can become a serious political problem.