I don't understand Jev. Its a generic classifier right? Like the classifiers we were building 15 years ago with random forrests and logistic regressions, but just generic. What's so revolutionary? And how can the accuracy be any better than a custom trained classifier that can be built in a day (an hour using Claude).
It is a classifier, that you don't have to train, that seems to work well, and they have made available by API.
One of the reasons for LLM success has not just been that they are "smart" but that they are smart while not requiring you to collect your own data and perform your own training.
There is also the marginal advantage of it not requiring a PhD to know how to use. But most companies can get a competent data scientist on board, so that is not a major problem, they could find someone to train their classifiers. But even a good data scientist can't make data appear out of nowhere. If you can get a pretty good result without needing data and training, for cheap, you're going to take it.
(Let's put aside the fact that almost every company using LLMs these days is irresponsibly NOT comparing the results with their own internal gold standard datasets for validation and calibration. They are just winging it. And Jev allows them to continue to do that.)
The core feature is good quality out-of-the-box domain and cross-domain calibrated probability you can use as a threshold.
SO you do not need to have YOUR data to train classifier. All you need is to benchmark it on your process/data to see that it really is good for your domain (as authors promise) and choose a threshold for acting.
Ahh, and the most important thing you need to do - design well the input context (state), questions with their context (instructions) and output type (because the same question asked as Noul (yes probability) or Choice with yes/no variants give different probability of yes.
I don’t think its much cheaper like luna is double the cost. and you can easily use luna to generate a training set and train your own model. I think the main things its faster so a lot of people go the Wow moment I can classify my whole db at once. Or this model decides so quickly. Speed is the key differentiator, it’s a strong one, always.
> Like the classifiers we were building 15 years ago with random forrests and logistic regressions
I think no one mentioned here, but the obvious difference is that Jev can spit out decisions directly from natural language input. None of these ML models could do that, and other than a full-fledged LLM (which is optimized for conversation and agentic tasks) or some classical NLP models (which underperform compared to LLMs, AFAIK), there is nothing right now that rivals Jev-like models.
Of course it's not perfect, but seems like the right step forward for quick classification/decision tasks based on natural language.
I was watching some youtube hype demo of it to pick items (each an emoji) from a pile according to questions. One of the questions was: "what can a magnet attract?" and it picked like 8 metal items, while a literal magnet was left unpicked. Ofc the video was too busy praising it to notice. Was all I needed to see
That it can make mistakes? That’s expected, it deal with plausibility like other models no? You would need to look at the actual response and the assigned probabilities to evaluate. Generally demos are not a great way to evaluate a technology, it’s a way to get hype but the next step is to actually look at the details
Yes, I don’t think hallucinations will ever go away. But because it doesn’t work with language the type of hallucinations aren’t too comparable to LLMs. But it is still a risk of course, whatever system you design should take that in consideration
Have you tried Jev and compared it against your alternative classifiers? Should take half an hour to do that, then you’ll have your answer (or someone who already did it can tell you here).
I did this yesterday with user data to find duplicates and determine which conflicting fields should win in a merge. Took me about an hour-ish to run the experiment. I provided a good amount of context for each pair. It was underwhelming / OK at best and didn't add any value to my existing merge workflow, so I threw it out after the experiment (I kept the findings though). It was quite easy to setup and use via openrouter and extremely cheap and fast.
For fun I also tried using it at my local LLM router (all of my prompts and responses go through it for personal analytics) to decide which model to route tasks to based complexity etc. Again, underwhelming for my purposes, and so I'm not using it.
Not sure what real world use case it's best at, but I agree that it's simple enough to implement it yourself and see if it fits the type of work you are doing.
I think your engineering process is fundamentally broken if you are generating PRs with 230 files changed so regularly that you need to bolt on more AI.
You’re solving the wrong problem.
Code is there to be read and understood by the human developers who come later.
The git history is a similar record, that's why the commits that make it to main (the squashed PRs) should cover one(ish) thing each and be self contained.
Something covering 230 files should be a mechanical change like running a linter or the AI is moving an API from one signature to another.
If an LLM generated a 230 file change they are also capable of going back and breaking it up.
One thing they are bad at is comments that are succinct since they almost only ever add words.
> Code is there to be read and understood by the human developers who come later.
Have you been under a rock in the last two years?? Code is written solely by AI now, and hence it needs to be understood by AI only. Humans can still give some feedback on architecture and high level design to feel important, but even that has its days counted already.
It's funny, because in my experience, the process often looks like:
1. Human gives high-level design.
2. Agent generates wrong code with misleading comments.
3. In further iterations, agent get mislead by said code and comments, ends up generating insane workarounds.
Sadly, the hype is real and that's what's happening in many places. Businesses are eager to bet on AI cause they'd been promised x100 productivity = fire 99 or 100 devs = huge profit.
I expect one day some slopware will succumb to one of these weird production bugs, no LLM will be able to fix it, and when the biz guys ask me for a fix estimate, I'm going to say "3 years".
I went all in on it about a year ago, and I've gotten the hype out of my system now. It's now officially a love/hate relationship. Like everything else that is a mainstay in my life.
It is inevitable. We (coders) already did it to other engineering disciplines where machines do most of the work, no human could possibly review or understand the actual details, and only verification ensures it is correct.
Sadly, I know (major) companies that insist it's the process that needs to be solved, because it improves velocity (for some definition of velocity that involves dropping pretty much all quality gates).
This is bananas. I've seen 25 year old software systems that haven't broken the 100k commits barrier. It feels like that ought to be enough for entire product lifecycles. What's going on that isn't simply wheelspinning?
This happens. I catch claude and codex committing broken code all the time, and then stacking micro fixes on top, still broken. You can go very long if you're in a harness until you realize it's just committing everything. Sometimes if you give it a goal, or a long task, this happens too, every little argument with itself, every finding, adjustment, is a new commit. None of them make sense, they happen anyway. One line nonsensical change, 20 line reassuring comment, committed.
Most PRs that are generated by agents today get YOLO merged because its hard for human mind to comprehend when your agent just suddenly shows up with 230 file changes. This is an attempt to reduce the mental burden by classifying each change in a review to P0, P1, P2. Only P0 are shown by default. The priorities are configurable. The diffs are also show using a natural language. The original code is one toggle away.
I yolo vibe develop, but how one end up with 210 files change commits? Unless one shooting a project, there is no way to end up with such scope?
I really don't get it.
"Plan out the change. Structure the plan so that it can be broken into individual PR's that each covers a single logical part of the change. Aim for less than 500 lines per PR unless the change is trivial and mechanical." (with a lower end model you need to be a lot more precise; with a frontier model you only really need to let it know you care about the PR size)
... in your prompts, coupled with gating PR creation on passing a stringent automated code review that also reviews it on whether or not it adheres to the rules you set regarding complexity of the PR, solves that problem without hiding the complexity.
If you have 230 file changes, that is a process failure unless they're purely obviously committing purely e.g. linter cleanups.
You don’t need classifier to filter noise. The noise is the same patterns repeated many times. I’ve got a tool does that to break down big diff and then feed to llm for grouping.
https://github.com/thepartly/differential
Jev has good use cases that require speed. But speed is not all. Tasks like code review need quality, which costs time. If you just want speed, you might as well go with a non-reasoning model like Haiku.
I'm going back to writing code by hand. AI can assist with debugging and research. I've had enough of slop. I've had enough of dealing with people wielding slop machines.
In the end... I don't feel better off. More tooling hasn't been the answer. It has only resulted in more complexity, less happiness, and about the same velocity as before we did agenetic engineering.
I don't understand Jev. Its a generic classifier right? Like the classifiers we were building 15 years ago with random forrests and logistic regressions, but just generic. What's so revolutionary? And how can the accuracy be any better than a custom trained classifier that can be built in a day (an hour using Claude).
I don't understand the hype.
> custom trained classifier
The "moat" is data. And compute. But mostly data.
It is a classifier, that you don't have to train, that seems to work well, and they have made available by API.
One of the reasons for LLM success has not just been that they are "smart" but that they are smart while not requiring you to collect your own data and perform your own training.
There is also the marginal advantage of it not requiring a PhD to know how to use. But most companies can get a competent data scientist on board, so that is not a major problem, they could find someone to train their classifiers. But even a good data scientist can't make data appear out of nowhere. If you can get a pretty good result without needing data and training, for cheap, you're going to take it.
(Let's put aside the fact that almost every company using LLMs these days is irresponsibly NOT comparing the results with their own internal gold standard datasets for validation and calibration. They are just winging it. And Jev allows them to continue to do that.)
The core feature is good quality out-of-the-box domain and cross-domain calibrated probability you can use as a threshold.
SO you do not need to have YOUR data to train classifier. All you need is to benchmark it on your process/data to see that it really is good for your domain (as authors promise) and choose a threshold for acting.
Ahh, and the most important thing you need to do - design well the input context (state), questions with their context (instructions) and output type (because the same question asked as Noul (yes probability) or Choice with yes/no variants give different probability of yes.
You don't need to train it, you can throw pretty much whatever at it and it'll work and it's extremely cheap. That's it.
I don’t think its much cheaper like luna is double the cost. and you can easily use luna to generate a training set and train your own model. I think the main things its faster so a lot of people go the Wow moment I can classify my whole db at once. Or this model decides so quickly. Speed is the key differentiator, it’s a strong one, always.
> Like the classifiers we were building 15 years ago with random forrests and logistic regressions
I think no one mentioned here, but the obvious difference is that Jev can spit out decisions directly from natural language input. None of these ML models could do that, and other than a full-fledged LLM (which is optimized for conversation and agentic tasks) or some classical NLP models (which underperform compared to LLMs, AFAIK), there is nothing right now that rivals Jev-like models.
Of course it's not perfect, but seems like the right step forward for quick classification/decision tasks based on natural language.
> What's so revolutionary?
Because it's all "AI" now <insert Ancient Aliens meme>
I also don't get the point. So what if the output format is different? It's still just as hallucination prone as before, right?
I was watching some youtube hype demo of it to pick items (each an emoji) from a pile according to questions. One of the questions was: "what can a magnet attract?" and it picked like 8 metal items, while a literal magnet was left unpicked. Ofc the video was too busy praising it to notice. Was all I needed to see
That it can make mistakes? That’s expected, it deal with plausibility like other models no? You would need to look at the actual response and the assigned probabilities to evaluate. Generally demos are not a great way to evaluate a technology, it’s a way to get hype but the next step is to actually look at the details
But isnt the whole point of Jev that "we trained it so you dont have to"? Yet it make such a simple mistakes still.
Yes, I don’t think hallucinations will ever go away. But because it doesn’t work with language the type of hallucinations aren’t too comparable to LLMs. But it is still a risk of course, whatever system you design should take that in consideration
Have you tried Jev and compared it against your alternative classifiers? Should take half an hour to do that, then you’ll have your answer (or someone who already did it can tell you here).
I did this yesterday with user data to find duplicates and determine which conflicting fields should win in a merge. Took me about an hour-ish to run the experiment. I provided a good amount of context for each pair. It was underwhelming / OK at best and didn't add any value to my existing merge workflow, so I threw it out after the experiment (I kept the findings though). It was quite easy to setup and use via openrouter and extremely cheap and fast.
For fun I also tried using it at my local LLM router (all of my prompts and responses go through it for personal analytics) to decide which model to route tasks to based complexity etc. Again, underwhelming for my purposes, and so I'm not using it.
Not sure what real world use case it's best at, but I agree that it's simple enough to implement it yourself and see if it fits the type of work you are doing.
Could you tell us? That would be simpler for everyone no?
I think your engineering process is fundamentally broken if you are generating PRs with 230 files changed so regularly that you need to bolt on more AI. You’re solving the wrong problem.
That really stood out.
Code is there to be read and understood by the human developers who come later.
The git history is a similar record, that's why the commits that make it to main (the squashed PRs) should cover one(ish) thing each and be self contained.
Something covering 230 files should be a mechanical change like running a linter or the AI is moving an API from one signature to another.
If an LLM generated a 230 file change they are also capable of going back and breaking it up.
One thing they are bad at is comments that are succinct since they almost only ever add words.
> Code is there to be read and understood by the human developers who come later.
Have you been under a rock in the last two years?? Code is written solely by AI now, and hence it needs to be understood by AI only. Humans can still give some feedback on architecture and high level design to feel important, but even that has its days counted already.
It's funny, because in my experience, the process often looks like:
1. Human gives high-level design. 2. Agent generates wrong code with misleading comments. 3. In further iterations, agent get mislead by said code and comments, ends up generating insane workarounds.
> human developers who come later
Tbf, once the codebase is slopped enough that becomes impossible and only LLM can come later.
Yeah, we should push back on this. We don't have to accept this outcome as if it's inevitable.
Sadly, the hype is real and that's what's happening in many places. Businesses are eager to bet on AI cause they'd been promised x100 productivity = fire 99 or 100 devs = huge profit.
I expect one day some slopware will succumb to one of these weird production bugs, no LLM will be able to fix it, and when the biz guys ask me for a fix estimate, I'm going to say "3 years".
I went all in on it about a year ago, and I've gotten the hype out of my system now. It's now officially a love/hate relationship. Like everything else that is a mainstay in my life.
It is inevitable. We (coders) already did it to other engineering disciplines where machines do most of the work, no human could possibly review or understand the actual details, and only verification ensures it is correct.
Why do you think coding is immune to this?
Call me old fashioned, but things were working just fine.
> no human could possibly review or understand the actual details,
What are you talking about? - The developers understood it. Did my car manufacturer do something to my discipline because I don't understand my car?
Look up Constraint-Driven Design or Declarative Engineering
It is.
Sadly, I know (major) companies that insist it's the process that needs to be solved, because it improves velocity (for some definition of velocity that involves dropping pretty much all quality gates).
I can confirm that our code quality at work is regressing and we are shipping less product features than before with AI.
Welcome to 2026.
My dev branch got broken when I rebased to main after a week of drift.
I had to bisect over around a million commits to the monorepo to find the culprit.
This is bananas. I've seen 25 year old software systems that haven't broken the 100k commits barrier. It feels like that ought to be enough for entire product lifecycles. What's going on that isn't simply wheelspinning?
This happens. I catch claude and codex committing broken code all the time, and then stacking micro fixes on top, still broken. You can go very long if you're in a harness until you realize it's just committing everything. Sometimes if you give it a goal, or a long task, this happens too, every little argument with itself, every finding, adjustment, is a new commit. None of them make sense, they happen anyway. One line nonsensical change, 20 line reassuring comment, committed.
Doing stuff at the 1e100.net scale.
> I had to bisect over around a million commits to the monorepo to find the culprit.
Thank god bisect is O(log n) at least...
Most PRs that are generated by agents today get YOLO merged because its hard for human mind to comprehend when your agent just suddenly shows up with 230 file changes. This is an attempt to reduce the mental burden by classifying each change in a review to P0, P1, P2. Only P0 are shown by default. The priorities are configurable. The diffs are also show using a natural language. The original code is one toggle away.
You can tell it not to do this?
> Most PRs that are generated by agents today get YOLO merged
Not quite. Ironically, they get PR-reviewed by someone else's agent. The humans in between are meat-proxies, pressing OK buttons.
I yolo vibe develop, but how one end up with 210 files change commits? Unless one shooting a project, there is no way to end up with such scope? I really don't get it.
Something like:
"Plan out the change. Structure the plan so that it can be broken into individual PR's that each covers a single logical part of the change. Aim for less than 500 lines per PR unless the change is trivial and mechanical." (with a lower end model you need to be a lot more precise; with a frontier model you only really need to let it know you care about the PR size)
... in your prompts, coupled with gating PR creation on passing a stringent automated code review that also reviews it on whether or not it adheres to the rules you set regarding complexity of the PR, solves that problem without hiding the complexity.
If you have 230 file changes, that is a process failure unless they're purely obviously committing purely e.g. linter cleanups.
It would be actually nice to build a simple classifier that marks noise from PRs (stuff like formatting changes for example)
For the rest, I wouldn't trust an AI to pick what's important.
We can already do AI reviews, and while helpful, we know we can't trust them too much
You don’t need classifier to filter noise. The noise is the same patterns repeated many times. I’ve got a tool does that to break down big diff and then feed to llm for grouping. https://github.com/thepartly/differential
Jev has good use cases that require speed. But speed is not all. Tasks like code review need quality, which costs time. If you just want speed, you might as well go with a non-reasoning model like Haiku.
Yey! Let's speedup project downfall to crap abyss even more!
You either care about project and do reviews. Or yolo vibe code. This middle ground has little sense.
jev is not good at reasoning, its a generic low-level classifier. for reviewing / classifying code, i would strongly suggest not using jev.
Nice idea. This could make large agent-generated PRs much easier to review.
I'm going back to writing code by hand. AI can assist with debugging and research. I've had enough of slop. I've had enough of dealing with people wielding slop machines.
In the end... I don't feel better off. More tooling hasn't been the answer. It has only resulted in more complexity, less happiness, and about the same velocity as before we did agenetic engineering.
confidence score really feels like real machine learning project which we test to check the confusion matrix
Slop projects to solve slop
The slop theory tells us that (slop * slop) = doubleslop.
You still review code?
Maybe improve your development process, learn how to use AI better so you don't have to.
Maybe only if you are writing OS or some very specific kind of software you have to do this.
For all the other kinds of software, you likely don't need to.
That's crazy and sooner or later will come back to bite you.
That's like telling a structural engineer that they don't need to review their plans.
If you don't like writing code or reviewing it, stop being a programmer, go become a manager or something
i read this as: "you are generating code wrong! you need to generate code better, so you don't even need to think about what you are generating!"
I have a better idea, what about stop spamming slop PRs to projects?
[dead]