I wish that "small" LLMs would stop being confidently very incorrect. Admittedly this is a bit of an intentionally esoteric test, but the confident way in which it presents a totally incorrect answer is a bit concerning.
"please write 250 words on the etymology and history of the word schlong"
The actual origin of the word is from middle high German and Yiddish-speaking Ashkenazi Jewish communities.
For comparison qwen 3.6 35B A3B does perfect on this and will give a solid description of the word's real origins and how it has made it into casual profanity/vulgarity as used in US English, and even mentions specific stand-up comedians and famous public figures of specific ethnic/religious origin in the US NE who introduced it into wider use.
Ask it for something that's not a narrow niche scientific or technical field, but something that would be less common to make it into a 20B size model, and see just how it does.
Looks promising though much like the bonsai tenary one it hallucinates knowledge quite aggressively. The online chat having search tools covers this up somewhat, but it's still there.
...from very unscientific casual vibes it does seem pretty good though considering the speed
Would also be curious what their search tool backend looks like - that too is very fast for very rapid multiple searches
Makes me wonder how much we can 'get away with' in terms of raw model capability, when you have tool access for anything specialised or specific.
I probably could make up a bit of an incorrect carrot cake recipe if asked on the spot, but with a Google search I can give you something far more robust.
Maybe we don't need a 'country of geniuses' in our pocket, but more a helpful assistant that can reasonably reason!
A benchmark table comparing to Qwen 3.5 35B-A3B seems strange when Qwen 3.6 35B-A3B has been out for some time and is significantly better.
I didn't notice the version difference when first reading the article! So this is a heads up to people like me.
I wish that "small" LLMs would stop being confidently very incorrect. Admittedly this is a bit of an intentionally esoteric test, but the confident way in which it presents a totally incorrect answer is a bit concerning.
"please write 250 words on the etymology and history of the word schlong"
https://pastes.io/uhshFgn4
The actual origin of the word is from middle high German and Yiddish-speaking Ashkenazi Jewish communities.
For comparison qwen 3.6 35B A3B does perfect on this and will give a solid description of the word's real origins and how it has made it into casual profanity/vulgarity as used in US English, and even mentions specific stand-up comedians and famous public figures of specific ethnic/religious origin in the US NE who introduced it into wider use.
Ask it for something that's not a narrow niche scientific or technical field, but something that would be less common to make it into a 20B size model, and see just how it does.
chat test link: https://chat.deepgrove.ai/
Looks promising though much like the bonsai tenary one it hallucinates knowledge quite aggressively. The online chat having search tools covers this up somewhat, but it's still there.
...from very unscientific casual vibes it does seem pretty good though considering the speed
Would also be curious what their search tool backend looks like - that too is very fast for very rapid multiple searches
Makes me wonder how much we can 'get away with' in terms of raw model capability, when you have tool access for anything specialised or specific.
I probably could make up a bit of an incorrect carrot cake recipe if asked on the spot, but with a Google search I can give you something far more robust.
Maybe we don't need a 'country of geniuses' in our pocket, but more a helpful assistant that can reasonably reason!
As of now, 3 of the 5 comments on this page are just 0-1 karma accounts high-fiving the article. Suspicious.
You can play with it online. I was pretty impressed with the speed/quality given it's size.
It's definitely not going to replace a larger frontier model, but it's worth keeping an eye on.
Edge is edging closer!
Super cool and a taste of what's to come with local AI becoming more accessible to low-end hardware.
This is cool!
i didnt think this was possible
this is quite smthn!
baller