Interestingly enough, both the GitHub repository + user apparently been deleted from GitHub. People who ran this project might want to take a look at the code/dependencies if they didn't do that beforehand.
I’ve been integrating benchmarks into OrcaBot either locally on the app or online at orcabot.com. Mostly as a way of giving an easy way to verify my own posted benchmarks. There is a template for running SlopCodeBench. Working on a TTS one next.
Wasn’t sure if there would be a lot of interest in this space. It can be costly but some provided such as OpenAI let you hammer their subscription with these.
Nice! I wanted something like this recently, but am trying to avoid vibe coding janky solutions to my problems, so it’s nice to see someone else taking this on.
Author here. I had a pile of models pulled locally and no way to answer "which of these is actually good, and what does it cost me in speed?" llama-bench gives you tok/s and nothing about output quality; lm-evaluation-harness gives you quality but isn't built around Ollama or LM Studio, which is what most people are actually running at home. So this measures both in one pass and puts them in one table.
The part that turned out to be harder than expected was deciding what the numbers mean:
tok/s sounds trivial until you pick a denominator. Prompt processing? Model load? I exclude both, use Ollama's server-side eval timing where it's reported, and fall back to timing the token stream client-side elsewhere. Those two paths aren't strictly comparable, which is why the methodology is in the README rather than a footnote.
Memory has no single honest answer. Runners that expose resident model size get reported directly; everything else gets a best-effort peak-RSS sample of the backend processes. Both are labelled as estimates rather than blended into one confident number.
Single-stream throughput hides half the picture on batching servers. A vLLM instance at concurrency 8 does far more aggregate work than the leaderboard's single-stream figure suggests, so batch throughput is a separate sweep reporting aggregate tok/s, speedup, and p95 latency. On a non-batching setup throughput stays flat while latency climbs, which is itself the useful result.
Quality grading is deterministic on purpose: exact numeric match, multiple choice, valid-JSON, regex, at temperature 0 with a fixed seed. There's an optional LLM-as-judge path for open-ended tasks but it's marked as a signal, not a score.
The honest limitation: the built-in suite is 31 tasks. That is a smoke test for "did this quantization break the model," not a leaderboard of record, and I make no contamination claims. It won't separate two models a few points apart, and it's English-only. If you need real eval rigor, use lm-evaluation-harness. You can point homebench at your own evals with a YAML task pack, which is what I'd suggest for anything you'd actually act on.
pip install homebench
homebench
Happy to hear where the methodology is wrong. That's the part I'm least sure about and the part that matters most.
The link is 404 but not just the repo - the author’s entire account appears to be gone
Interestingly enough, both the GitHub repository + user apparently been deleted from GitHub. People who ran this project might want to take a look at the code/dependencies if they didn't do that beforehand.
FYI the GitHub URL is 404ing
I’ve been integrating benchmarks into OrcaBot either locally on the app or online at orcabot.com. Mostly as a way of giving an easy way to verify my own posted benchmarks. There is a template for running SlopCodeBench. Working on a TTS one next.
Wasn’t sure if there would be a lot of interest in this space. It can be costly but some provided such as OpenAI let you hammer their subscription with these.
Source available: https://github.com/Hyper-Int/OrcaBot
Nice! I wanted something like this recently, but am trying to avoid vibe coding janky solutions to my problems, so it’s nice to see someone else taking this on.
Author here. I had a pile of models pulled locally and no way to answer "which of these is actually good, and what does it cost me in speed?" llama-bench gives you tok/s and nothing about output quality; lm-evaluation-harness gives you quality but isn't built around Ollama or LM Studio, which is what most people are actually running at home. So this measures both in one pass and puts them in one table.
The part that turned out to be harder than expected was deciding what the numbers mean:
tok/s sounds trivial until you pick a denominator. Prompt processing? Model load? I exclude both, use Ollama's server-side eval timing where it's reported, and fall back to timing the token stream client-side elsewhere. Those two paths aren't strictly comparable, which is why the methodology is in the README rather than a footnote.
Memory has no single honest answer. Runners that expose resident model size get reported directly; everything else gets a best-effort peak-RSS sample of the backend processes. Both are labelled as estimates rather than blended into one confident number.
Single-stream throughput hides half the picture on batching servers. A vLLM instance at concurrency 8 does far more aggregate work than the leaderboard's single-stream figure suggests, so batch throughput is a separate sweep reporting aggregate tok/s, speedup, and p95 latency. On a non-batching setup throughput stays flat while latency climbs, which is itself the useful result.
Quality grading is deterministic on purpose: exact numeric match, multiple choice, valid-JSON, regex, at temperature 0 with a fixed seed. There's an optional LLM-as-judge path for open-ended tasks but it's marked as a signal, not a score.
The honest limitation: the built-in suite is 31 tasks. That is a smoke test for "did this quantization break the model," not a leaderboard of record, and I make no contamination claims. It won't separate two models a few points apart, and it's English-only. If you need real eval rigor, use lm-evaluation-harness. You can point homebench at your own evals with a YAML task pack, which is what I'd suggest for anything you'd actually act on.
Happy to hear where the methodology is wrong. That's the part I'm least sure about and the part that matters most.It would be great if there was a much more thorough example output in the README, including e.g. Qwen 3.8 27B (once it comes out).
[flagged]