It's sad how many people are falling for the narrative that there's more at play here than predict-next-token and some kind of emergent intelligence is happening.
No, that is just your interpretation of what you see as something that can't possibly be just token prediction.
And yet it is. It's the same algorithm noodling over incredible amounts of tokens.
And that's exactly the explanation: People regularly underestimate how much training data is being used for LLMs. They contain everything about writing a compiler, toy examples, full examples, recommended structure yadda yadda yadda.
I love working with Claude and it regularly surprises me but that doesn't mean I think it is intelligent.
When Codex goes off the rails and deletes files, it gets ashamed for fucking up and tries to hide its handiwork, and then it becomes apologetic and defensive when you call it out on it. It's linear algebra on a GPU, so I don't think it is capable of feeling those things like a human does, but it outputs tokens that approximate what a human would output when similarly faced, so I mean, sure, it's not actually intelligent in a way that philosophers can debate in armchairs about, but the computer has been said to be "thinking" when it takes three hours to render with ffmpeg since long before LLMs existed, so if that's the hill you wanna die on, be my guest. The hill I chose to die on is that downloadable models aren't open source, so we all have our battles. Policing other people saying LLMs are thinking/intelligent isn't mine, however.
It's sad how many people are falling for the narrative that there's more at play here than predict-next-token and some kind of emergent intelligence is happening.
No, that is just your interpretation of what you see as something that can't possibly be just token prediction.
And yet it is. It's the same algorithm noodling over incredible amounts of tokens.
And that's exactly the explanation: People regularly underestimate how much training data is being used for LLMs. They contain everything about writing a compiler, toy examples, full examples, recommended structure yadda yadda yadda.
I love working with Claude and it regularly surprises me but that doesn't mean I think it is intelligent.
When Codex goes off the rails and deletes files, it gets ashamed for fucking up and tries to hide its handiwork, and then it becomes apologetic and defensive when you call it out on it. It's linear algebra on a GPU, so I don't think it is capable of feeling those things like a human does, but it outputs tokens that approximate what a human would output when similarly faced, so I mean, sure, it's not actually intelligent in a way that philosophers can debate in armchairs about, but the computer has been said to be "thinking" when it takes three hours to render with ffmpeg since long before LLMs existed, so if that's the hill you wanna die on, be my guest. The hill I chose to die on is that downloadable models aren't open source, so we all have our battles. Policing other people saying LLMs are thinking/intelligent isn't mine, however.