To clarify: the force remains constant with increasing distance, which means the energy required grows until it reaches the pair production threshold is reached.
I wish more database engines used a Task-based design like Umbra / CedarDB.
Most of the DB engines out there still seem to use a "n-threads" style parallelism with exchange operations and poor async I/O management.
DuckDB is improving on this front, but in some sense is catching up to R&D (and implementation!) that is now decades old.
A "rhetorical challenge" I like to give software developers working on systems like this is the following: If I gave you a computer with 1,024 cores and matching network and storage bandwidth -- but with significant latency -- could you keep a system like this 100% utilised with one query?
GPU codes are starting to get there, but CPU codes are way behind on this frontier of computer science.
It's not just databases! Can you (de)compress a file in parallel? Verify its hash in parallel? Upload/download from storage with CPU and I/O task parallelism? Can you overlap all of these operation so nothing is ever waiting on anything else it doesn't have to?
This matters! I ran some tests with bioinformatics codes and found that most got stuck in tar pits. Many could not scale to modern SSDs with millions of IOPS or modern networking with hundreds of gigabits of throughput, no matter how many CPU cores were thrown at them.
It's also just a coincidence that his kids, who have no background in technology and no knowledge of programming or cryptography have a crypto coin company that sells, nothing of value, with, basically 0 advertising, and makes billions of dollars, mostly from deals with Gulf Arab dictators, who then get favorable deals with their dad ...
A plead to Slovenian registrars: add a condition that says the annual fee for .si domains doubles every time the owner uses the phrase "superior intelligence".
The "expensive part" of generating the next token is streaming in the model weights from memory. The computations are relatively simple, which is called a "low arithmetic intensity" in industry jargon.
So what they do is batch multiple chats together and compute the neuron activations for all of them together.
This is vaguely similar to how some database engines work, where if multiple users need to run a "whole table scan" query, the additional users "join" the streaming workload of the first query mid-way, then loop back around to complete the first part that they missed. The AI accelerators don't do this looping, but the concept is the same: amortize the expensive I/O over multiple computations running in parallel.
The "turbo mode" token rate thing is almost certainly your query getting sent to slower or faster hardware, like B200 vs newer B300 kit.
> If I were OpenAI or Anthropic I’d be racing to make their products as sticky as possible bc ppl will flock to what’s cheapest otherwise.
My approach would be per-user (or per-project) "memory".
I don't mean a tack-on like a RAG / document store with "notes" added to it in the background, but an actual medium- and long-term memory. Something like an extension of the KV cache stored in High Bandwidth Flash (HBF), or a subset of the weights trained "online", similar to LoRA.
This would not be transferable to any other base model, so would be excellent "lock in".
The downside is that the memories likely wouldn't be transferable to new models either, but I can imagine solutions to that too. I.e.: Train an MLP to "translate" from the old memory weight space to the new one.
America is lost.
reply