On: Llama.cpp supports Vulkan. why doesn't Ollama?

[chown]

I totally agree with this. I wanted to make it really easy for non-technical users with an app that hid all the complexities. I basically just wanted to embed the engine without making users open their terminal, let alone make them configure. I started with llama.cpp amd almost gave up on the idea before I stumbled upon Ollama, which made the app happen[1]

There are many flaws in Ollama but it makes many things much easier esp. if you don’t want to bother building and configuring. They do take a long time to merge any PRs though. One of my PRs has been waiting for 8 months and there was this another PR about KV cache quantization that took them 6 months to merge.

[1]: https://msty.app

[zozbot234 → chown]

> They do take a long time to merge any PRs though.

I guess you have a point there, seeing as after many months of waiting we finally have a comment on this PR from someone with real involvement in Ollama - see https://github.com/ollama/ollama/pull/5059#issuecomment-2628... . Of course this is very welcome news.

[wkat4242 → zozbot234]

It's not really welcome news, he is just saying they're putting it on the long finger because they think other stuff is more important. He's the same guy that kept ignoring the KV cache quant merge.

And the actual patch is tiny..

I think it's about time for a bleeding-edge fork of ollama. These guys are too static and that is not what AI development is all about.

[zozbot234 → wkat4242]

He specifically says that they're reworking the Ollama server implementation in order to better support other kinds of models, and that such work has priority and is going to be a roadblock for this patch. This is not even news to those who were following the project, and it seems reasonable in many ways - users will want Vulkan to work across the board if it's made available at all, not for it to be limited to the kinds of models that exist today.

[smcleod → chown]

That qkv PR was mine! Small world.