Zero Data Retention is supposed to mean something. In practice, these prompts were retained, shared with other companies, and read by people.
The first bit of advice is on point:
Host it yourself, if you can afford it. This is the only option where nobody else gets to read your prompts, but the hardware isn’t cheap. A Mac Studio with 512 GB of memory starts at $22,299, NVIDIA’s DGX Spark now costs $6,950, a DGX Station starts at about $105,000, and an eight-GPU B300 server lists for $570,000 or more. For most teams, this only makes sense for the most sensitive work.
Despite being an NLP researcher, I never used LLMs for coding until this summer. Glad I exclusively used locally hosted models. My M5 Ultra 512GB just arrived and I'm hoping it turns out to be a wise decision for my startup.
https://www.dragos.com/blog/ai-supply-chain-security-canary-...