Hi HN, I've spent the better part of the last year deploying AI inference systems for both personal and work projects. I've noticed a ton of "gotchas" and footguns throughout the process that make doing it right a pain, so I built "Batteries Included AI Deployment". Not a creative name but it does what it says on the tin.
In short, it's a template for deploying AI inference APIs with FastAPI.
In long, it uses Docker to encapsulate (almost) the entire development and deployment process. The repo includes:
1. A way to download and cache models straight from huggingface
2. A way to expose those cached models via a FastAPI server endpoint
3. A docker configuration that exposes a `debugpy` port so that you can debug your application within a container
4. A way to run tests
5. A way to debug tests (using `debugpy` as mentioned above)
6. A way to run pre-commits on staged files
7. A way to manually run pre-commits on all code in your repository
8. CI steps via GitHub Actions
9. Full Observability with a Grafana Dashboard
10. Metrics via Prometheus
11. Tracing via Tempo
12. Logs via Loki
13. GPU monitoring via DCGM
14. CD via GitHub actions and a `post-receive` hook on the server
15. Alerts that email you when something goes wrong in production
I say "almost" because you still need a way to attach to the debugger port from outside the docker container and there's some one-time configurations that need to be set up manually, but not anything beyond that.
I'd love to hear any feedback you might have :)
