For inference, if you have a supported card (or probably architecture if you are on Linux and can use HSA_OVERRIDE_GFX_VERSION), then you can probably run anything with (upstream) PyTorch and transformers. Also, compiling llama.cpp is has been pretty trouble-free for me for at least a year.
(If you are on Windows, there is usually a win-hip binary of llama.cpp in the project's releases or if things totally refuse to work, you can use the Vulkan build as a (less performant) fallback).
Having more options can't hurt, but ROCm 5.4.2 is almost 2 years old, and things have come a long way since then, so I'm curious about this being published freshly today, in October 2024.
BTW, I recently went through and updated my compatibility doc (focused on RDNA3) w/ ROCm 6.2 for those interested. A lot has changed just in the past few months (upstream bitsandbytes, upstream xformers, and Triton-based Flash Attention): https://llm-tracker.info/howto/AMD-GPUs
i also have been playing with inference on the amd 7900xtx, and i agree. there are no hoops to jump through these days. just make sure to install the rocm version of torch (if using a1111 or similar, don't trust requirements.txt), as shown clearly on the pytorch homepage. obsidian is a similar story. hip is straightforward, at least on arch and ubuntu (fedora still requires some twiddling, though). i didn't realize xformers is also functional! that's good news.
I think fazkan was confused about which repo you were talking about. For the llm-tracker doc, that's something I maintain. It's based on stuff I test but if you want to submit a PR or issue w/ info in a way that I can verify then I'm happy to add a Docker section.
haha, I was a bit confused, but I was referring to this one https://github.com/slashml/amd_inference. But the comment applies to other repos as well, do open issues in them, helps the maintainers prioritize features.
I also had to go to therapy to cure myself of the misunderstanding that data scientists and machine learning folks are software engineers, and expecting the same work product from those disparate audiences only raises your blood pressure
Expectation management is a huge part of any team/organization, I think
They can be the same or different, given how you define them. People throw these words around with little thought, especially ones superficial to or outside the field.
I wouldn't disparage an entire field for lack of a clear definition in the buzzwords people use to refer to it.
While I see where you are coming from, these are the types of comments that keep people from sharing their code, contributing to OSS or continuing to program in general.
It seems to use an old, 2 year old version of ROCm (5.4.2) which I'm doubtful would support my RX 7900 XTX. I personally found it easiest to just use the latest `rocm/pytorch` image and run what I need from there
The RX 7900 XTX (gfx1100) was first enabled in the math libraries (e.g. rocBLAS) for ROCm 5.4, but I don't think the AI libraries (e.g. MIOpen) had it enabled until ROCm 5.5. I believe the performance improved significantly in later releases, as well.
On Ubuntu 24.04 (and Debian Unstable¹), the OS-provided packages should be able to get llama.cpp running on ROCm on just about any discrete AMD GPU from Vega onwards²³⁴. No docker or HSA_OVERRIDE_GFX_VERSION required. The performance might not be ideal in every case⁵, but I've tested a wide variety of cards:
# install dependencies
sudo apt -y update
sudo apt -y upgrade
sudo apt -y install git wget hipcc libhipblas-dev librocblas-dev cmake build-essential
# ensure you have permissions by adding yourself to the video and render groups
sudo usermod -aG video,render $USER
# log out and then log back in to apply the group changes
# you can run `rocminfo` and look for your GPU in the output to check everything is working thus far
# download a model, build llama.cpp, and run it
wget https://huggingface.co/TheBloke/dolphin-2.2.1-mistral-7B-GGUF/resolve/main/dolphin-2.2.1-mistral-7b.Q5_K_M.gguf?download=true -O dolphin-2.2.1-mistral-7b.Q5_K_M.gguf
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
git checkout b3267
HIPCXX=clang-17 cmake -H. -Bbuild -DGGML_HIPBLAS=ON -DCMAKE_HIP_ARCHITECTURES="gfx803;gfx900;gfx906;gfx908;gfx90a;gfx1010;gfx1030;gfx1100;gfx1101;gfx1102" -DCMAKE_BUILD_TYPE=Release
make -j16 -C build
build/bin/llama-cli -ngl 32 --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -m ../dolphin-2.2.1-mistral-7b.Q5_K_M.gguf --prompt "Once upon a time"
I'd suggest RDNA 3, MI200 and MI300 users should probably use the AMD-provided ROCm packages for improved performance. Users that need PyTorch should also use the AMD-provided ROCm packages, as PyTorch has some dependencies that are not available from the system packages. Still, you can't beat the ease of installation or the compatibility with older hardware provided by the OS packages.
¹ https://lists.debian.org/debian-ai/2024/07/msg00002.html
² Not including MI300 because that released too close to the Ubuntu 24.04 launch.
³ Pre-Vega architectures might work, but have known bugs for some applications.
⁴ Vega and RDNA 2 APUs might work with Linux 6.10+ installed. I'm in the process of testing that.
⁵ The version of rocBLAS that comes with Ubuntu 24.04 is a bit old and therefore lacks some optimizations for RDNA 3. It's also missing some MI200 optimizations.
I was able to install (AMD provided) ROCm and Ollama on Ubuntu 22.04.5 with an RX 7900 XTX with no real problems to speak of, and I can execute LLMs using Ollama on ROCm just fine. Take that FWIW.
The Radeon Pro W6800, W7800 or W7900 would be the standard answer. A hacker-spirited alternative would be to purchase a used MI50, MI60 or MI100 and 3d print a fan adapter. There are versions of all of those cards with 32GB of VRAM and they can be found on ebay for between 350 USD and 1200 USD. Plus twenty bucks for a fan adapter and a fan.
Those old gfx906 or gfx908 cards are more competitive for fp64 than for low-precision AI workloads, but they have the memory and the price is right. I'm not sure I would recommend the hacker approach to the average user, but it is what I've done for some of the continuous integration servers I host for the Debian project.
It sort of depends on how you define "consumer friendly prices". AFAIK, in the $1000 - "slightly over or under $1000" range, 24GB is all you can get. But there are Radeon Pro boards with 32GB or 48GB of RAM for various prices between around $2000 to about $3500. So not "cheap" but possibly within reach for a serious hobbyist who doesn't mind spending a little bit more.
It has been like 8 months since I got Ryzen 8700G with NPU just for the purpose of inferencing NN, and so far only acceleration I'm getting is through vulkan on iGPU, not NPU (I'm using Linux only). On the bright side, with 64GB of RAM had no isues with trying models over 32GB. Kudos to llama.cpp for supporting vulkan backend!
You should have ROCm/HIP support on the iGPU as well, be sure to compile llama.cpp w/ the LLAMA_HIP_UMA=1 flag. If you take a look at https://github.com/amd/RyzenAI-SW you can see there's a fair amount of software to play with on the NPU now, but Phoenix is only 16 TOPS, so I've never bothered testing it.
So no doubt modern software is ridiculously bloated, but ROCm isn't just a GPU driver. It includes all sorts of tools and libraries as well.
By comparison, if you go and download the CUDA toolkit as a single file, you get a download file that's over 4GB, so quite a bit larger than the download size you quoted. I haven't checked how much that expands to (it seems the ROCm install has a lot of redundancy given how well it compresses), but the point is, you get something that seems insanely large either way.
The biggest one just to pick on one is hipblaslt is "a library that provides general matrix-matrix operations. It has a flexible API that extends functionalities beyond a traditional BLAS library, such as adding flexibility to matrix data layouts, input types, compute types, and algorithmic implementations and heuristics." https://github.com/ROCm/hipBLASLt
There are mostly GPU kernels that by themselves aren't so big, but for every single operation x every single supported graphics architecture, eg:
My understanding is that ROCm contains all included kernels for each supported architecture, so it would have (made up):
-- matrix multiply 2048x2048 for Navi 31,
-- same for Navi 32,
-- same for Navi 33,
-- same for Navi 21,
-- same for Navi 22,
-- same for Navi 23,
-- same for Navi 24, etc.
-- matrix multiply 4096x4096 for Navi 31,
-- ...
Correct. Although, you wouldn't find Navi 22, 23 or 24 in the list because those particular architectures are not supported. Instead, you'd see Vega 10, Vega 20, Arcturus, Aldebaran, Aqua Vanjaram and sometimes Polaris.
We're working on a few different strategies to reduce the binary size. It will get worse before it gets better, but I think you can expect significant improvements in the future. There are lots of ways to slim the libraries down.
Wait, looking at that link I don't see how it avoids downloading CUDA or ROCM. Do you use MLIR to compile to GPU without using the vendor provided tooling at all?
> Of course, much of that is auto-generated header files... A large portion of it with AMD continuing to introduce new auto-generated header files with each new generation/version of a given block. These verbose header files has been AMD's alternative to creating exhaustive public documentation on their GPUs that they were once known for.
What's the best bang-for-your-buck AMD GPU these days? I just bought 2 used 3090s for $750ish refurb'd on eBay. Curious what others are using for running LLMs locally.
I bought an MI100 recently for $650. 32GB of HBM2 and it performs around around 0-5% faster than a 3090 on the default flash attention 2 benchmarks. Performance on actual applications can be mixed though, as many are not well optimised for CDNA's matrix cores - even where work has been done for RDNA, which is not that often, it doesn't necessarily carry over. It's also frustrating when efforts to improve performance get turned back by maintainers: llama.cpp closing PR for flash attention on AMD because the requisite (header-only) lib is supposedly adding an unneeded dependency (https://github.com/ggerganov/llama.cpp/pull/7011).
There's also a few tricks/updates I'd like to try which may improve performance, e.g. hipblaslt support being added next rocm release - of course these are "maybes".
To give you a rough idea of practical performance, default SDXL with xformers is around 4.5-5it/s (between 3090 and 4090 from my understanding), and exllamav2 with qwen 72B at 3bpw is around 7t/s (slower than a 3090, though a 3090 has to use a lower precision to fit).
As others have pointed out, I can't really see what this project offers for AMD users over existing options like llama.cpp, exllamav2, mlc-ai, etc. Most projects work relatively easily these days.
Personal experience: It's not even worth it. AMD (i)GPU breaks with every pytorch, ROCm, xformers, or ollama updates. You'll sleep more compfortably at night.
When dealing with ROCM, it's critical that once you have a working configuration, you freeze everything in place (except your application). Docker is one way to achieve this if your host machine is subject to kernel or package updates
I don't really have any problem with ROCm these days, although I only use system packages. It used to be quite wonky though, and I've totally given up on custom ROCm installs.
I got my radeon pro vii for €300 new. Was not a bad deal IMO especially since it comes with HBM2 and has the same memory bandwidth as the 4090 (1TB/s). It's got only 16GB though.
I doubt it, but the 3090 is a four year old card which means it might have a lot of mileage from the previous owner. A lot of them are from mining rigs.
You can do inference from a Docker container, just as you'd do it with NVidia. OpenAI runs a K8s cluster doing this. I have personally only worked with NVidia, but the docs are present for AMD too.
Like anything AI and AMD, you need the right card(s) and rocm version along with sheer dumb luck to get it working. AMD has Docker images with rocm support, so you could merge your app in with that as the base layer. Just pass through the GPU to the container and you should get it working.
It might just be the software in a Docker image, but it removes a variable I would otherwise have to worry about during deployment. It literally is inference on AMD with Docker, if that's what you meant.
Docker became part of the standard toolkit for ML because deploying Python that links to underlying system libraries is a gong show unless you ship that layer too.
Even Docker doesn't guarantee reproducible results due to sensitivity towards host GPU drivers, and ML frontends/integrations bringing their own "helpful" newby-friendly all-in-one dependency checks and updater services.
Yeah, they're using docker to wrap up the software packages, which is what Docker is used for. I don't understand why that confuses you or what you think Docker is otherwise used for.
I'm pretty comfortable with Docker/cgroups/namespaces, I have quite a deep understanding of it. But I read "Docker-based inference" like you literally took Docker code to... do inference? The wording in my opinion doesn't make much sense. It's like saying, I don't know, "Flatpak-based inference" or "SSD-based inference". Semantics.
If you're interested in how much the AMD graphics cards cost compared to the NVidia ones, I have https://gpuquicklist.com/ which gives you a quick table view of lowest prices available on Amazon that I can find. </ selfpromotion>
There are some magic environment variables you want to set to get ROCM to work with this technically unsupported APU:
HSA_OVERRIDE_GFX_VERSION=9.0.0
HSA_ENABLE_SDMA=0
Performance is not great, but slightly better than running inference on the CPU, with the bonus that your CPU is essentially free for other tasks even while running LLMs.
is anyone using the new HX370 based laptops for any LLM work ? i mean the ipex-llm libraries of Intel's new Lunar Lake is already supporting Llama 3.2 (https://www.intel.com/content/www/us/en/developer/articles/t...), but AMD's new Zen5 chips dont seem to be much active here.
I'm all for having more open source projects, but I do not see how it can be useful in this ecosystem, especially for people with newer AMD GPUs (not supported in this project) which are already supported in most popular projects?
we have vllm in certin production instances, it is a pain for most non-nvidia related architectures. A bit of digging around and we realized that most of it is just a wrapper on top of pytorch function calls. If we can do away with batch processing with vllm supports, we can be good, this is what we did here.
driver mismatch issues, we mostly use publicly available instances, so the drivers change as the instances change, according to their base image. Not saying it won't work, but it was more painful to figure out vllm, than to write a simple inference script and do it ourselves.
(If you are on Windows, there is usually a win-hip binary of llama.cpp in the project's releases or if things totally refuse to work, you can use the Vulkan build as a (less performant) fallback).
Having more options can't hurt, but ROCm 5.4.2 is almost 2 years old, and things have come a long way since then, so I'm curious about this being published freshly today, in October 2024.
BTW, I recently went through and updated my compatibility doc (focused on RDNA3) w/ ROCm 6.2 for those interested. A lot has changed just in the past few months (upstream bitsandbytes, upstream xformers, and Triton-based Flash Attention): https://llm-tracker.info/howto/AMD-GPUs