On: From the creator of Redis; run LLM locally with ds4

[wg0]

Can someone explain me what expertise (domain knowledge) one needs to be able to write such model specific inference engine?

The other inference engine are also model by model with a huge switch statement deciding which part to load for which model or are they very generic?

[epolanski → wg0]

Antirez understood the fundamentals of how LLMs work, and read the inference code (or summaries of it via ai).

But what really made the difference was his understanding of hardware and systems programming in general and low level or architectural tricks to pull.