On: Breaking Quadratic Barriers: A Non-Attention LLM for Ultra-Long Context Horizons

[cubefox]

> "hundreds of thousands to potentially millions of tokens" - that's the same order as current commercial LLMs.

Yes, but those are all relying on proprietary company secrets, while this is an open research paper. Besides, only Gemini so far has a context window of more than a million tokens.

[littlestymaar → cubefox]

Llama 4 Scout has it also, and is an open weight LLM, unfortunately it is also disappointing at pretty much any context length…