On: Extending the context length to 1M tokens

[anon291]

If we did to cats what he did to GPT models that would be animal abuse.

That is to say, if we want to extend this analogy, the model is 'killed' after each round. This is hardly a criticism of the underlying technology.

Going back to feeding the entire input. That is not really true. There are a dozen ways to not do that these day.