On: How we made a text-to-speech model respond in sub-50 ms

[narrationbox]

Haven't read the full report yet, just a quick question. Are your numbers for cold start without pre fill or is it after warmed cache?

[toebee → narrationbox]

We do graph capture etc at startup (same as vLLM) but this model variant doesn’t require prefix caching - the prefix is just 10 tokens.