> workloads can be scheduled on low priority batches
I feel like there's enormous potential still uncovered in the coding agent space for such highly-discounted "spot" style capacity. Give me a way to have agents continuously crawl over my codebase looking for bugs, come up with test cases that prove the bug, open a PR. I don't care if it takes 12 hours to find something that would take 5 minutes if prompted with "on-demand" / "high-priority" requests, because I'm not prompting for it directly, I'm not aware of how much time it takes, all I see is continuous improvement.
Isn’t the KV cache a huge component of the overall costs, and wouldn’t this make this problematic? Cache writes are an expensive part of LLM tokens, even more so for caches that have a long expiration time.
Cache is a huge component of "high priority" costs because it essentially forces the GPU to be idle (holding earlier prompt text in GPU memory) while waiting for newer input tokens. When an agentic session is active, this is of course much more preferable compared to loading in hundreds of thousands of input tokens again and again on different GPUs, but "walking away from the laptop" while waiting for the cache to expire is something that royally sucks today for inference providers. (Caveat: this is my layperson's understanding, I don't work in the space, I'm likely not up to date on best-practice caching practices)
"Low-priority" inference tasks like that, though, would never need to be cached. Again, who cares if it takes 12 hours? If you didn't initiate the prompt, then what do you care? You just randomly get notified when something is already ready for you to take a look at.
Isn’t caching much more nuanced, with different tiers (DRAM, local disk, remote disk, etc), and isn’t filling these caches something that happens asynchronously (ie while the GPU is still working on the previous batch) ?
I think reality is much more nuanced than what you’re sketching.
But that's my point - throw the "background code review" stuff out of cache, or into object storage, or whatever, just something to draw upon to fill inference batches opportunistically.
I mean, at the end of the day, either inference providers are having difficulty filling batches, in which case there's opportunity on the table to somehow fill the batches with discounted, time-insensitive, "background review" style work, or they don't have that difficulty, in which case it's a moot point. Exact cache architectures are beside the point.
I feel like there's enormous potential still uncovered in the coding agent space for such highly-discounted "spot" style capacity. Give me a way to have agents continuously crawl over my codebase looking for bugs, come up with test cases that prove the bug, open a PR. I don't care if it takes 12 hours to find something that would take 5 minutes if prompted with "on-demand" / "high-priority" requests, because I'm not prompting for it directly, I'm not aware of how much time it takes, all I see is continuous improvement.