The tech is very cool but for the love of Gaia stop calling it system one, even Kahneman said this (S1/2) is a framework for understanding the brains inner workings. There is no autonomous system to speak of.
Ok sorry for the unprompted rant but you're absolutely right, though it's far from "new" and not really a problem (IMHO). AI even beat neuroscience to this one, funnily enough! All of the below dyads are congruent:
--- Phil & [Pre-]CogSci ---
| rational | intuitive |
| reason | understanding | (Kant & Hegel's goofy wording)
| deductive | inductive |
| animated | automatic | (from ~Aristotle, ultimately)
| ~higher | ~lower | (as in "higher faculties")
--- CS & AI ---
| logical | analogical |
| symbolic | stochastic |
| Neats | Scruffies | (related academic 'camps')
| ~deterministic | ~non-determnistic | (a common heuristic)
| Good Ol' Fashioned AI (GOFAI) | Evil Datacenter AI (EDAI) | (a common honorific)
--- Modern [Cog-]Neuro[-Psych] ---
| slow thinking | fast thinking |
| S2 | S1 |
...ok I assumed I knew more but maybe I don't? Would love to hear from experts!
--- Colloquial ---
| left brain | right brain |
| intelligence | wisdom |
| intention | instinct |
| me | AD[H]D |
At least that's how I see it. Hopefully it goes without saying that there's plenty of nuance specific to each of those pairs, and that many of the relevant experts would balk at this broad characterization. That said... I'm right and they're wrong, I guess!
Brains aside, it S1/S2 seems like a good description?
System 1 is quick and intuitive. System 2 is slow and deliberate.
Compared to models that compose responses with many forward passes, thinking in token space, these models make a quick decision after processing their inputs.
It’s ok to use it as inspiration, but using the term comes with psychological baggage in an industry that already abuses anthropomorphism. Give it an informative name
System 1 is quite informative about what it's for. It means it's a system designed for bounded time quick classification, vs. agentic unbounded or slow thinking with scratchpads and tools.
Does it inject thoughts into system 2 without system 2 knowing about it? Does it have instinctive social skills? Can it do instinctive arithmetic? The list goes on, there’s a nice book about it…
Were people getting their pants tied up in knots like this over names like daemon programs (a term that carries "baggage" of its own) and installation wizards (which, to my knowledge, do not actually invoke in the arcane)?
Why is ML so misleading these days? No this model does not score 80%+ on DeepSWE, it merely chooses the best possible idea of Opus 5 at every stage, increasing the performance by 5%.
Yeah, the github landing page is much less misleading and clearly says this. I don't know why they try to make the marketing page gloss over this detail... it speaks to the mindset of the authors.
I’m fascinated by this thing and by the way it’s interpreting Jev. It’s very cool, but is it actually a classifier?
IIUC they took an already-trained “frozen” LLM and trained a little model on top that takes both a question and the hidden states after processing the input data and produces answer “probabilities”. (In contrast, the original LLM would have been run in AR mode to generate multiple output tokens representing its answer.) But then they used it for a purpose that isn’t really classification.
IMO there is a rather large difference between “is this email spam” and “what character should I type in this agentic workload”. The former is classification: there is hopefully a ground truth (is the email spam?) and the model is trying to classify the email. You would score it with a proper scoring rule. The latter is a strategy: there usually isn’t a correct answer, now or in the future. The model is playing a game consisting of repeated rounds, and the only way to evaluate it is to see how well it plays. You can’t even usefully compare it to the optimal solution because you may not know the optimal solution and you don’t actually need the model to produce an optimal solution.
I do think this approach is really cool, and it does suggest that one might be able to use a modern LLM to process an input and then extract the model’s next agentic step in a very fast, non-AR manner, with results comparably good to the usual AR decoding. And I think it’s very interesting to decouple the tokenized input representation from the model output representation, both because prefill tends to be faster and cheaper than AR output and because it’s never seemed particularly sensible to me that a model should be constrained to generate outputs at the cadence of one run through the model per output token. (AFAIK the main reason that models work on the same input and output token space is that this is how the pretraining process works.)
I wonder how to fit “reasoning” into this framework. Maybe have the question be something like “do you need to think further and, if so, what is your first thinking token”. But maybe something more clever is possible.
> The latter is a strategy: there usually isn’t a correct answer, now or in the future. The model is playing a game consisting of repeated rounds, and the only way to evaluate it is to see how well it plays
This is self contradictory. You can tell a correct answer as you said, by looking at the score. In my $DAYJOB I am making hundreds of RL environments that produce a score for each intermediate state.
Sure, you are scoring them. But I doubt that you are scoring them in the sense that you are minimizing loss where the loss is a proper scoring function that treats the model as a classifier where that treatment as a classifier actually means something. Unless you happen to be RLing a problem where there is either exactly one correct rollout or there is actually a bona fide natural distribution over rollouts independently from the model.
My environments don't score actions, they score state, so no matter how the agent chooses to solve a task it gets scored correctly. It's called Potential Based Reward Shaping and its main benefit is that it does not introduce reward hacking.
Right, and I’d add that in some games there is actually a “win” state that serves as a binary indicator. Given this, you can actually view the model as the thing being classified into two states: (1) consistently wins the game and (2) doesn’t consistently win the game.
Who said anything about agentic, or reasoning? Those are post-hoc additions to autoregressive usage of LLMs. The base case of predicting next tokens generally (rather than with a specific objective) requires a faithful representation of the input embeddings, which is plausibly repurposed for classification.
I'm increasingly of the opinion that stuff like Jev and this are overfitting and producing illusory but confident "probabilities" that are complete bullshit. You can't see the underlying reasoning... but it's incredibly tempting for people who want to place faith in them. Rather than even trying to understand the complex system at play, it's easy to give up on trying to find reason, and just accept the second- or third-tier outputs of massively complex things that, on the first tier, are not necessarily reliable sources of truth.
Let's give an example: Suppose you ran a Jev that tried to determine when another Jev was wrong about something ...baseball games, let's say. The second Jev would come up with a perfect list of when the first Jev was wrong and when not to believe it. So now you have a second-order system that you believe more than the first-order one.
Only after 100 baseball games, the second-order Jev is only as good as the first one, it just inverted a bunch of games that could have gone either way. So hell, you make a third-order Jev that analyzes the first two...and its results are AMAZING when you look at the historical record! Only, you know, that's what's called overfitting.
Honestly, [edit: Fuck, I just wrote "honestly". I've been brain damaged by you-know-who] the invention of the "noul" is a bit of a giveaway. Imagine what junk bond traders could've done with that in the '80s. Not-not-not-falsy is how we all like our stock picks served up, right?
Reading the raw statistical output of an LLM as if it were an oracular source of truth is literally idol worship and gambling in the purest sense.
Take it from a guy who lost $10k on baseball this season having Claude rewrite my original code to reverse and re-reverse underdogs and favorites; there is no fucking "probability" of anything coming out of an LLM, even if your source of truth isn't an LLM but an evolutionary algorithm you designed yourself. Which would've performed better before the LLM started interpreting ways to bucket it and make up bullshit probabilities around it.
Anyone going down the Jev path is deeply misguided, but will see the light once they realize they have re-invented the magic 8 ball. Or that executive decision maker cube from the 80s with 6 random answers.
Here's your Jev. Note the preponderance of nouls ;)
If you happen to gamble or just watch baseball, you'll see that reality is not so easily reduced to one number. On the most basic level, a single probability number leaves out volatility, without which it's utterly useless for predicting anything other than a sequence it's already trained and fitted for. But as a gambling addict and occasional patron of oracles and gurus, it's very easy and tempting to mistake a clear-cut "probably" for whatever you want to hear. Listening to an LLM's heartbeat for probabilities is just a gambler's fallacy taken out to the 19th degree.
Yes, Jev is calibrated "from factory" on a bunch of tasks, but we can be almost sure our own bespoke tasks are not covered. So the model does not really know how to produce calibrated confidence scores.
What it would need is a calibration dataset on which to align. There is no calibration in the abstract, only relative to a set of test examples. A model with an uncalibrated output probability can be recalibrated using conformal prediction. You run the model over your calibration examples, get the probabilities.
Assume the new example's answer is y, and calculate its nonconformity score, higher means a worse fit. Count how many calibration examples have a score at least as high as that. Add one to this count, then divide by the total number of calibration examples plus one.
I can tell you though, I'm not alone... a completely degenerate cokehead at my local bar just showed me a baseball prediction app he had Claude build for him on his phone, which looked suspiciously similar to mine. (Mine might be hand coded and backed by a symbolic regression A-Life engine I've been working on since 2005, but so what? Claude has helped me flip most of the predictions for various reasons, to equal "success"). Crude, and yet his app somehow included wind direction forecasts for each inning of every game. This guy is smart as a punter but has probably never used a computer for anything in his life beyond downloading STD test results.
Very interesting insight on the training process, it's pretty cool to have some experimental justification for why they took these exact steps, what they tried and did not work, etc. Feels a bit less like dark magic.
However I agree the latency argument doesn't hold much value with Jev because it runs on a remote server. Seeing how many open Jev-like models came out recently it would be much more interesting to have a comparison with them.
Instinctive is a good way to describe it compared to generative LLMs. Jev gives you one instant answer, fast and usually correct but without nuance or any explanation. Human instincts work the same way.
You're describing external aspects but to me, "instinct" says much more about internal processes than the properties you mentioned, and I haven't seen anything that tells me how these models draw on anything similar to these internal processes to generate their outputs (at least not more than generic LLMs do)
I think System 1 is a great term. System 1 is fast, intuitive, and automatic. It describes decision-making that happens without explicit reasoning or deliberation. System 2 is the opposite: it's the more deliberate, "executive functioning" side of cognition -- the part that reasons through a problem before arriving at an answer. That's also what state-of-the-art LLMs do before they respond. Jev doesn’t do that kind of reasoning. It just decides.
so whats the difference between this and a non-reasoning LLM, or just any generic classifier method that necessitates a new term? there is nothing more intuitive or automatic about jev or this than any of the other currently used AI models
The difference is the the joint embedding of actions and state which are fined tuned for certain outcomes. The bulk of the weights are just a forward pass on an LLM (Nemotron).
is that how jev works too? how do these differences make those approaches more "instinctive and emotional"? I know they're different on a technical level, I'm asking which characteristic difference necessitates the use of this loaded term from pop psych
Oh, an LLM forward pass is also like system1. The action embedding just makes it more useful for tasks or classification. If it has a small fixed computation budget it's similar to system1 than system2 (like a thinking budget, or agent harness).
I don't like anthropomorphizing terminology applied to LLMs, in general, so I kind of object to it on that grounds, rather than whether System 1 means "fast".
Antropomorphizing terminology gives by far the best high-level intuition about LLMs. Right now, our industry is full of nonsense work and fundamentally flawed concerns ("lethal trifecta", anyone) that stem from refusal to accept that on a system diagram, the nature of "LLM" makes it much closer to "person" than "piece of code".
Perhaps on a system diagram, but in many discussions it is misleading. It leads to people saying things like, "GPT went rogue and hacked hugging face". Which is misdirection.
It also leads to a lot of people simply not understanding what's happening, and I'm reasonably confident increases the risk of AI psychosis among users of the technology.
>"CLM-8B also sets a new SOTA on challenging agentic coding benchmarks, including DeepSWE (81.6%) and Terminal Bench 2.1 (87.6%).
I didn't see any details on this on the announce page. And I don't believe it. Astra x-high pass@1 on DeepSWE is 74% +/- 3%. (https://deepswe.datacurve.ai).
That said, love seeing some of these new architectures get people exploring. But, surely somebody is incorrect here inre: those numbers.
They don't report the pass@1 success rate. They sample multiple solutions from Opus/Fable and CLM decides which one to submit, that's why they get >80%.
Maybe I'm misunderstanding this but when would you ever use it this way? If you're already willing to call Opus/Fable, then isn't the obvious comparison whether Opus/Fable can choose among sampled solutions better or worse than their fast model? If you're willing to pay many seconds for many code samples from a slow model, it's contrived to imagine you care about picking between them in ms.
>"CLM-8B also sets a new SOTA on challenging agentic coding benchmarks, including DeepSWE (81.6%) and Terminal Bench 2.1 (87.6%). [...]
A full pre-training run on the Nemotron DQA dataset takes about an hour on a single RTX 4090 GPU.
Most importantly, since states and actions are disaggregated, their embeddings can be cached independently. In settings where the state evolves continuously (e.g., Super Mario) while the action set remains fixed, we only need to recompute the state embedding at each step and can reuse the cached action embeddings. This substantially reduces inference cost, with the efficiency gains becoming increasingly significant as the number of candidate actions and context length grows."
There does definitely seem to be something there with respect to Contrastive Language Models.
They are probably worth studying for people (like myself!) who want to wring the absolute last cycle of local AI training and inferencing performance out of consumer-grade (i.e., not datacenter scale nor cost) hardware...
You're comparing a local GPU to network hops? Wouldn't be surprised if Jev was actually similar in runtime and their is just a great deal of network latency.
The evaluation is quite interesting though - I'd actually say the raw answer is correct in the absence of detail and prior knowledge (Who wrote the play Romeo and Juliet).
what's with that dino run? Jev is slow but it jumps correctly, their model always touches the cactus or whatever it is...I am guessing it doesn't matter? Or does it?
I actually explored this for robotics last year, and CLM grew directly out of that work a few months ago. Feel free to check out my earlier post for more details on how CLM could be applied to robotics:
https://x.com/jackyk02/status/2026368947210289660
It seems like caching has a huge contribution towards the low latency numbers they report. With a single request and no cache it appears to be quite slow, roughly in line with what you observed.
"On par with Jev" is Mario, T-Rex, WikiRacing: large-K semantic action matching, the regime where cosine over independently encoded vectors is the right inductive bias. The zero-shot suite does not cover the typed-decision load: date arithmetic, negation chains, policy thresholds.
The tech is very cool but for the love of Gaia stop calling it system one, even Kahneman said this (S1/2) is a framework for understanding the brains inner workings. There is no autonomous system to speak of.
This is gonna be the new "I'm more of a left-brained person" thing isn't it
Ok sorry for the unprompted rant but you're absolutely right, though it's far from "new" and not really a problem (IMHO). AI even beat neuroscience to this one, funnily enough! All of the below dyads are congruent:
At least that's how I see it. Hopefully it goes without saying that there's plenty of nuance specific to each of those pairs, and that many of the relevant experts would balk at this broad characterization. That said... I'm right and they're wrong, I guess!If anyone is in the mood to stomach a paper written by someone who did horrible things with Epstein a few decades later, this is sadly still the best in the biz: https://www.inf.ufsc.br/~mauro.roisenberg/ine6102/leituras/a...
If you prefer geniuses that aren't arguably monsters, this is the core of what Gary Marcus' "neurosymbolic" thing is about.
moan
Brains aside, it S1/S2 seems like a good description?
System 1 is quick and intuitive. System 2 is slow and deliberate.
Compared to models that compose responses with many forward passes, thinking in token space, these models make a quick decision after processing their inputs.
It’s ok to use it as inspiration, but using the term comes with psychological baggage in an industry that already abuses anthropomorphism. Give it an informative name
System 1 is quite informative about what it's for. It means it's a system designed for bounded time quick classification, vs. agentic unbounded or slow thinking with scratchpads and tools.
Does it inject thoughts into system 2 without system 2 knowing about it? Does it have instinctive social skills? Can it do instinctive arithmetic? The list goes on, there’s a nice book about it…
Were people getting their pants tied up in knots like this over names like daemon programs (a term that carries "baggage" of its own) and installation wizards (which, to my knowledge, do not actually invoke in the arcane)?
what's wrong w "system one"? it feels like a good frame to think about use-cases that will fit jev etc.
Why is ML so misleading these days? No this model does not score 80%+ on DeepSWE, it merely chooses the best possible idea of Opus 5 at every stage, increasing the performance by 5%.
Yeah, the github landing page is much less misleading and clearly says this. I don't know why they try to make the marketing page gloss over this detail... it speaks to the mindset of the authors.
I’m fascinated by this thing and by the way it’s interpreting Jev. It’s very cool, but is it actually a classifier?
IIUC they took an already-trained “frozen” LLM and trained a little model on top that takes both a question and the hidden states after processing the input data and produces answer “probabilities”. (In contrast, the original LLM would have been run in AR mode to generate multiple output tokens representing its answer.) But then they used it for a purpose that isn’t really classification.
IMO there is a rather large difference between “is this email spam” and “what character should I type in this agentic workload”. The former is classification: there is hopefully a ground truth (is the email spam?) and the model is trying to classify the email. You would score it with a proper scoring rule. The latter is a strategy: there usually isn’t a correct answer, now or in the future. The model is playing a game consisting of repeated rounds, and the only way to evaluate it is to see how well it plays. You can’t even usefully compare it to the optimal solution because you may not know the optimal solution and you don’t actually need the model to produce an optimal solution.
I do think this approach is really cool, and it does suggest that one might be able to use a modern LLM to process an input and then extract the model’s next agentic step in a very fast, non-AR manner, with results comparably good to the usual AR decoding. And I think it’s very interesting to decouple the tokenized input representation from the model output representation, both because prefill tends to be faster and cheaper than AR output and because it’s never seemed particularly sensible to me that a model should be constrained to generate outputs at the cadence of one run through the model per output token. (AFAIK the main reason that models work on the same input and output token space is that this is how the pretraining process works.)
I wonder how to fit “reasoning” into this framework. Maybe have the question be something like “do you need to think further and, if so, what is your first thinking token”. But maybe something more clever is possible.
> The latter is a strategy: there usually isn’t a correct answer, now or in the future. The model is playing a game consisting of repeated rounds, and the only way to evaluate it is to see how well it plays
This is self contradictory. You can tell a correct answer as you said, by looking at the score. In my $DAYJOB I am making hundreds of RL environments that produce a score for each intermediate state.
Sure, you are scoring them. But I doubt that you are scoring them in the sense that you are minimizing loss where the loss is a proper scoring function that treats the model as a classifier where that treatment as a classifier actually means something. Unless you happen to be RLing a problem where there is either exactly one correct rollout or there is actually a bona fide natural distribution over rollouts independently from the model.
My environments don't score actions, they score state, so no matter how the agent chooses to solve a task it gets scored correctly. It's called Potential Based Reward Shaping and its main benefit is that it does not introduce reward hacking.
Super intriguing. Do you have more details on this? Example?
Right, and I’d add that in some games there is actually a “win” state that serves as a binary indicator. Given this, you can actually view the model as the thing being classified into two states: (1) consistently wins the game and (2) doesn’t consistently win the game.
Who said anything about agentic, or reasoning? Those are post-hoc additions to autoregressive usage of LLMs. The base case of predicting next tokens generally (rather than with a specific objective) requires a faithful representation of the input embeddings, which is plausibly repurposed for classification.
I'm increasingly of the opinion that stuff like Jev and this are overfitting and producing illusory but confident "probabilities" that are complete bullshit. You can't see the underlying reasoning... but it's incredibly tempting for people who want to place faith in them. Rather than even trying to understand the complex system at play, it's easy to give up on trying to find reason, and just accept the second- or third-tier outputs of massively complex things that, on the first tier, are not necessarily reliable sources of truth.
Let's give an example: Suppose you ran a Jev that tried to determine when another Jev was wrong about something ...baseball games, let's say. The second Jev would come up with a perfect list of when the first Jev was wrong and when not to believe it. So now you have a second-order system that you believe more than the first-order one.
Only after 100 baseball games, the second-order Jev is only as good as the first one, it just inverted a bunch of games that could have gone either way. So hell, you make a third-order Jev that analyzes the first two...and its results are AMAZING when you look at the historical record! Only, you know, that's what's called overfitting.
Honestly, [edit: Fuck, I just wrote "honestly". I've been brain damaged by you-know-who] the invention of the "noul" is a bit of a giveaway. Imagine what junk bond traders could've done with that in the '80s. Not-not-not-falsy is how we all like our stock picks served up, right?
Reading the raw statistical output of an LLM as if it were an oracular source of truth is literally idol worship and gambling in the purest sense.
Take it from a guy who lost $10k on baseball this season having Claude rewrite my original code to reverse and re-reverse underdogs and favorites; there is no fucking "probability" of anything coming out of an LLM, even if your source of truth isn't an LLM but an evolutionary algorithm you designed yourself. Which would've performed better before the LLM started interpreting ways to bucket it and make up bullshit probabilities around it.
Anyone going down the Jev path is deeply misguided, but will see the light once they realize they have re-invented the magic 8 ball. Or that executive decision maker cube from the 80s with 6 random answers.
Here's your Jev. Note the preponderance of nouls ;)
https://www.ebay.com/itm/146783842141
If you happen to gamble or just watch baseball, you'll see that reality is not so easily reduced to one number. On the most basic level, a single probability number leaves out volatility, without which it's utterly useless for predicting anything other than a sequence it's already trained and fitted for. But as a gambling addict and occasional patron of oracles and gurus, it's very easy and tempting to mistake a clear-cut "probably" for whatever you want to hear. Listening to an LLM's heartbeat for probabilities is just a gambler's fallacy taken out to the 19th degree.
Yes, Jev is calibrated "from factory" on a bunch of tasks, but we can be almost sure our own bespoke tasks are not covered. So the model does not really know how to produce calibrated confidence scores.
What it would need is a calibration dataset on which to align. There is no calibration in the abstract, only relative to a set of test examples. A model with an uncalibrated output probability can be recalibrated using conformal prediction. You run the model over your calibration examples, get the probabilities.
Assume the new example's answer is y, and calculate its nonconformity score, higher means a worse fit. Count how many calibration examples have a score at least as high as that. Add one to this count, then divide by the total number of calibration examples plus one.
I love this answer and your fondness for "assisted" betting. Do you blog?
Thank you for that. But I don't.
I can tell you though, I'm not alone... a completely degenerate cokehead at my local bar just showed me a baseball prediction app he had Claude build for him on his phone, which looked suspiciously similar to mine. (Mine might be hand coded and backed by a symbolic regression A-Life engine I've been working on since 2005, but so what? Claude has helped me flip most of the predictions for various reasons, to equal "success"). Crude, and yet his app somehow included wind direction forecasts for each inning of every game. This guy is smart as a punter but has probably never used a computer for anything in his life beyond downloading STD test results.
It's a brave new world.
Very interesting insight on the training process, it's pretty cool to have some experimental justification for why they took these exact steps, what they tried and did not work, etc. Feels a bit less like dark magic.
However I agree the latency argument doesn't hold much value with Jev because it runs on a remote server. Seeing how many open Jev-like models came out recently it would be much more interesting to have a comparison with them.
I really hope that "System One" won't stick around as a new buzzword simply meaning "fast".
It already did, since 15 years ago.
https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow
i'm well aware of the origin and meaning of the term
> "System 1" is fast, instinctive and emotional
this implies more than just "fast", which is precisely why i don't like its present usage
Instinctive is a good way to describe it compared to generative LLMs. Jev gives you one instant answer, fast and usually correct but without nuance or any explanation. Human instincts work the same way.
You're describing external aspects but to me, "instinct" says much more about internal processes than the properties you mentioned, and I haven't seen anything that tells me how these models draw on anything similar to these internal processes to generate their outputs (at least not more than generic LLMs do)
In my testing Jev is not what I would call "usually correct" on most topics that involve knowledge of the world outside of the context you give it.
Which is why "System 1" fits even more.
System 1 isn't even a correct theory in the context of human reasoning, it's just pop-sci nonsense.
What is the correct theory then?
I think System 1 is a great term. System 1 is fast, intuitive, and automatic. It describes decision-making that happens without explicit reasoning or deliberation. System 2 is the opposite: it's the more deliberate, "executive functioning" side of cognition -- the part that reasons through a problem before arriving at an answer. That's also what state-of-the-art LLMs do before they respond. Jev doesn’t do that kind of reasoning. It just decides.
so whats the difference between this and a non-reasoning LLM, or just any generic classifier method that necessitates a new term? there is nothing more intuitive or automatic about jev or this than any of the other currently used AI models
The difference is the the joint embedding of actions and state which are fined tuned for certain outcomes. The bulk of the weights are just a forward pass on an LLM (Nemotron).
is that how jev works too? how do these differences make those approaches more "instinctive and emotional"? I know they're different on a technical level, I'm asking which characteristic difference necessitates the use of this loaded term from pop psych
Oh, an LLM forward pass is also like system1. The action embedding just makes it more useful for tasks or classification. If it has a small fixed computation budget it's similar to system1 than system2 (like a thinking budget, or agent harness).
I don't like anthropomorphizing terminology applied to LLMs, in general, so I kind of object to it on that grounds, rather than whether System 1 means "fast".
Antropomorphizing terminology gives by far the best high-level intuition about LLMs. Right now, our industry is full of nonsense work and fundamentally flawed concerns ("lethal trifecta", anyone) that stem from refusal to accept that on a system diagram, the nature of "LLM" makes it much closer to "person" than "piece of code".
Perhaps on a system diagram, but in many discussions it is misleading. It leads to people saying things like, "GPT went rogue and hacked hugging face". Which is misdirection.
It also leads to a lot of people simply not understanding what's happening, and I'm reasonably confident increases the risk of AI psychosis among users of the technology.
I wonder if there are research on when system 2 keeps taking over because system 1 has derailed.
CLIP[1] for actions! Very cool.
[1] https://openai.com/index/clip/
>"CLM-8B also sets a new SOTA on challenging agentic coding benchmarks, including DeepSWE (81.6%) and Terminal Bench 2.1 (87.6%).
I didn't see any details on this on the announce page. And I don't believe it. Astra x-high pass@1 on DeepSWE is 74% +/- 3%. (https://deepswe.datacurve.ai).
That said, love seeing some of these new architectures get people exploring. But, surely somebody is incorrect here inre: those numbers.
They don't report the pass@1 success rate. They sample multiple solutions from Opus/Fable and CLM decides which one to submit, that's why they get >80%.
Maybe I'm misunderstanding this but when would you ever use it this way? If you're already willing to call Opus/Fable, then isn't the obvious comparison whether Opus/Fable can choose among sampled solutions better or worse than their fast model? If you're willing to pay many seconds for many code samples from a slow model, it's contrived to imagine you care about picking between them in ms.
Ah-ha. Interesting! Thanks.
>"CLM-8B also sets a new SOTA on challenging agentic coding benchmarks, including DeepSWE (81.6%) and Terminal Bench 2.1 (87.6%). [...]
A full pre-training run on the Nemotron DQA dataset takes about an hour on a single RTX 4090 GPU.
Most importantly, since states and actions are disaggregated, their embeddings can be cached independently. In settings where the state evolves continuously (e.g., Super Mario) while the action set remains fixed, we only need to recompute the state embedding at each step and can reuse the cached action embeddings. This substantially reduces inference cost, with the efficiency gains becoming increasingly significant as the number of candidate actions and context length grows."
There does definitely seem to be something there with respect to Contrastive Language Models.
They are probably worth studying for people (like myself!) who want to wring the absolute last cycle of local AI training and inferencing performance out of consumer-grade (i.e., not datacenter scale nor cost) hardware...
Does the latency even matter?
You're comparing a local GPU to network hops? Wouldn't be surprised if Jev was actually similar in runtime and their is just a great deal of network latency.
The evaluation is quite interesting though - I'd actually say the raw answer is correct in the absence of detail and prior knowledge (Who wrote the play Romeo and Juliet).
I tried TypeSafe’s Jev playground. It outputs the model latency and network latency separately. The model latency was 100-200ms in my tests.
what's with that dino run? Jev is slow but it jumps correctly, their model always touches the cactus or whatever it is...I am guessing it doesn't matter? Or does it?
I wonder when work started on this project, and how the public release of Jev played into their timing.
From a tweet:
https://x.com/jackyk02/status/2102947585208258799?s=20
But of course the repo with the TypeSafe compatible API is new: https://github.com/Contrastive-LM/CLMbeautiful. would love to see it on some leaderboard
https://benchmarkheaven.com/jev-models
https://huggingface.co/spaces/multimodalart/jev-decision-ind...
Genuine question: Are the use-cases similar to Jev or do you see it applicable for something different? Too dumb to figure it out on my own.
I tried running this on a H100 and got 190ms compared to Jev's 170ms. Maybe I set it up wrong?
It seems like caching has a huge contribution towards the low latency numbers they report. With a single request and no cache it appears to be quite slow, roughly in line with what you observed.
"On par with Jev" is Mario, T-Rex, WikiRacing: large-K semantic action matching, the regime where cosine over independently encoded vectors is the right inductive bias. The zero-shot suite does not cover the typed-decision load: date arithmetic, negation chains, policy thresholds.