Thanks for the comment! Fair point. We do not publish that curve for this model, and the HF model card should say so more clearly.
We did run an internal, unpublished FP16-anchored ladder on Qwen3-8B previously using the same pipeline:
MATH-500, MBPP+, commonsense-6
2.00 bpw: 68.8, 76.5, 73.1
2.44 bpw: 70.8, 76.5, 74.3
2.75 bpw: 72.8, 78.8, 74.4
3.00 bpw: 74.2, 82.0, 75.1
FP16: 75.6, 84.1, 74.1
At 2 bits, the 8B model is about 9% below FP16 on MATH-500 and MBPP+, but only about 1% below on commonsense. The loss is concentrated in generative math and code, and largely returns as bits increase.
That is one reason this release is a 35B MoE at 2 bits rather than an 8B model: there is more redundancy to exploit. On the 35B, the worst regression is LiveCodeBench at -4.4 points, or -6.6% relative. The other five benchmarks are within run-to-run noise. You can find more details on benchmark results and their harness on our HF model card page.
We do not have a BF16 arm for the 35B because it is about 70 GB and does not fit our hardware (we did this with one 4090, one L40, and tested on a bunch of other RTX GPUs :), so FP8 is the reference.
The 35B bit-width ladder is measured in KL divergence against FP8:
2.09 bpw: 0.1041
3.09 bpw: 0.0921
4.13 bpw: 0.0892
The 2-to-3-bit improvement is statistically significant. The 3-to-4-bit improvement is not.
How does it compare against Unsloth's UD-Q2_K_XL which has the same size? Ie, why pick yours over theirs?
I'm missing an accuracy vs bits graph here.
Cool to see this running but if it's at 10%+ loss vs 16bit that's less cool.
Hard to understand that story from what's published here.
Thanks for the comment! Fair point. We do not publish that curve for this model, and the HF model card should say so more clearly.
We did run an internal, unpublished FP16-anchored ladder on Qwen3-8B previously using the same pipeline:
2.00 bpw: 68.8, 76.5, 73.12.44 bpw: 70.8, 76.5, 74.3
2.75 bpw: 72.8, 78.8, 74.4
3.00 bpw: 74.2, 82.0, 75.1
FP16: 75.6, 84.1, 74.1
At 2 bits, the 8B model is about 9% below FP16 on MATH-500 and MBPP+, but only about 1% below on commonsense. The loss is concentrated in generative math and code, and largely returns as bits increase.
That is one reason this release is a 35B MoE at 2 bits rather than an 8B model: there is more redundancy to exploit. On the 35B, the worst regression is LiveCodeBench at -4.4 points, or -6.6% relative. The other five benchmarks are within run-to-run noise. You can find more details on benchmark results and their harness on our HF model card page.
We do not have a BF16 arm for the 35B because it is about 70 GB and does not fit our hardware (we did this with one 4090, one L40, and tested on a bunch of other RTX GPUs :), so FP8 is the reference.
The 35B bit-width ladder is measured in KL divergence against FP8:
2.09 bpw: 0.1041
3.09 bpw: 0.0921
4.13 bpw: 0.0892
The 2-to-3-bit improvement is statistically significant. The 3-to-4-bit improvement is not.
[dead]