Yesterday I released the first version of hyper-optimized Qwen3.8-27B inference engine for a RTX 3090, reaching 82 tps on single request and 672 peak.
Over the last 24 hours I've been exploring how to improve it further (without a quality degredation).
Yesterday we had these 3 optimizations:
- fp8 kv cache
- lm_head int8
- embed_tokens in8
Now now I added:
- fp16 recurrent state
- int8 activations, MLP
- int8 activations on everything
- draft_sample_method=probablistic which samples from the MTP distribution rather than taking the argmax) - does not work for greedy sampling.
For single user mode, the user can enable ctx=fast, which uses FlashAttention, bf16 kv, 64k context and gives the user 99 tps, but at the tradeoff of less context.
Also the prefill is now up to 1812 tps at 1k and 1000 tps at 100k context - an increase of about 50% to 25%.
Yesterday I released the first version of hyper-optimized Qwen3.8-27B inference engine for a RTX 3090, reaching 82 tps on single request and 672 peak.
Over the last 24 hours I've been exploring how to improve it further (without a quality degredation).
Yesterday we had these 3 optimizations: - fp8 kv cache
- lm_head int8
- embed_tokens in8
Now now I added: - fp16 recurrent state
- int8 activations, MLP
- int8 activations on everything
- draft_sample_method=probablistic which samples from the MTP distribution rather than taking the argmax) - does not work for greedy sampling.
For single user mode, the user can enable ctx=fast, which uses FlashAttention, bf16 kv, 64k context and gives the user 99 tps, but at the tradeoff of less context.
Also the prefill is now up to 1812 tps at 1k and 1000 tps at 100k context - an increase of about 50% to 25%.
I've updated the repo will everything you need to start flying: https://github.com/syv-ai/qwen38-27b-rtx3090