41 points | by Maverick617 3 hours ago ago
4 comments
> llama.cpp via Vulkan (AMD / Intel / NVIDIA) or CPU fallback
I got excited about someone paying attention to intel. Oh well.
From what I've seen, Vulkan adds a lot of overhead on Intel hardware.
I didn't see any benchmarks against vllm, sglang, exllama, etc
Since this is basically a wrapper around libllama.so, I would assume that the performance is roughly the same as llama.cpp upstream.
> llama.cpp via Vulkan (AMD / Intel / NVIDIA) or CPU fallback
I got excited about someone paying attention to intel. Oh well.
From what I've seen, Vulkan adds a lot of overhead on Intel hardware.
I didn't see any benchmarks against vllm, sglang, exllama, etc
Since this is basically a wrapper around libllama.so, I would assume that the performance is roughly the same as llama.cpp upstream.