On: Making AMD GPUs competitive for LLM inference (2023)

[idonotknowwhy]

Most people running local inference do so thorough quants with llamacpp (which runs on everything) or awq/exl2/mlx with vllm/tabbyAPI/lmstudio which are much faster to than using pytorch directly