presumably it's the reverse engineered server that has most of the work put into it, and one would hope that's what is going to be released if the developer decides to
which is important to bear in mind if people are introducing a "drop earliest messages" sliding window for context management in a "chat-like" experience.
once you're at that context limit and start dropping the earliest messages, you're guaranteeing every message afterwards will be a cache miss.
a simple alternative approach is to introduce hysteresis by having both a high and low context limit. if you hit the higher limit, trim to the lower. this batches together the cache misses.
if users are able to edit, remove or re-generate earlier messages, you can further improve on that by keeping track of cache prefixes and their TTLs, so rather than blindly trimming to the lower limit, you instead trim to the longest active cache prefix. only if there are none, do you trim to the lower limit.
I thought OpenAI would still handle case? Their cache would work up to the end of the file and you would then pay for uncached tokens for the user's question. Have I misunderstood how their caching works?
i've heard good things about using the 1 euro filter for user input related tasks, where you're trying to effectively remove noise, but also keep latency down.
That sounds very interesting. I've been needing a filter to deal with noisy A/D conversions for pots in an audio project. Noise on a volume control turns into noise on the output, and sounds horrible, but excessive filtering causes unpleasant latency when using the dials.
PHP does but the types actually mean something. If your types can be stripped out to make the program run, I have a hard time believing that there is any optimization occurring there.
for me, my desktop and laptop are the main go-to. the mobile is an extra device with different, more specific use cases
and so I've been a little disappointed with how these devices keep getting bigger and bigger. I was pretty happy with the size of the Pixel 3
I think I like to be able to access the whole screen comfortably with one hand, not fumbling it about. easy to manipulate, easy to pocket.
the Pixel 8 shrunk a bit over its predecessors so I nabbed that, and it's probably at or just over the limit for me, size wise
