That question sounds to me like “have any of you switched away from myspace?”
I last used dropbox ~15 years ago. Since then I used S3, Google Drive, Synology NAS, and iCloud. I don’t need to do any syncing across devices: if I need a file I know where to find it (gmail if it’s a document, icloud if it’s an image/video). I still back everything up to Synology.
What do you mean? We want images and text to live in the same latent space, and be represented by similar vectors if the two correlate. How else would you want to do it?
Not yet. It will probably start later this year: instead of hiring a junior swe, companies will buy gpt-5 or claude-4 subscriptions and ask senior engineers to deliver more.
Mass layoffs are still 2-3 years away, but I’m expecting this time next year the team I’m on will shrink because of AI.
> Ilya Sutskever, co-founder of AI labs Safe Superintelligence (SSI) and OpenAI, told Reuters recently that results from scaling up pre-training - the phase of training an AI model that use s a vast amount of unlabeled data to understand language patterns and structures - have plateaued.
OpenAI took a bullet for the team, by perhaps scaling the model to something bigger than the 1.6T params GPT4 possibly had and basically telling its competitors its not gonna be worth scaling much beyond those number of params in GPT4, without a change in the model architecture
1. RAG: A simple model looks at the question, pulls up some associated data into the context and hopes that it helps.
2. Self-RAG: The model "intentionally"/agentically triggers a lookup for some topic. This can be via a traditional RAG or just string search, ie. grep.
3. Full Context: Just jam everything in the context window. The model uses its attention mechanism to pick out the parts it needs. Best but most expensive of the three, especially with repeated queries.
Aider uses kind of a hybrid of 2 and 3: you specify files that go in the context, but Aider also uses Tree-Sitter to get a map of the entire codebase, ie. function headers, class definitions etc., that is provided in full. On that basis, the model can then request additional files to be added to the context.
I'm still not sure I get the difference between 1 and 2. What is "pulls up some associated data into the context" vs ""intentionally"/agentically triggers a lookup for some topic"?
1. Tends to use embeddings with a similarity search. Sometimes called "retrieval". This is faster but similarity search doesn't alway work quite as well as you might want it to.
2. Instead lets the agent decide what to bring into context by using tools on the codebase. Since the tools used are fast enough, this gives you effectively "verified answers" so long as the agent didn't screw up its inputs to the tool (which will happen, most likely).
Does it make sense to use vector search for code? It's more for vague texts. In the code relevant parts can be found by exact name match. (in most cases. both methods aren't exclusive)
Vector search for code can be quite interesting - I've used it for things like "find me code that downloads stuff" and it's worked well. I think text search is usually better for code though.
I'm still reading the paper, but my main question is how slow is the model compared to LLM of the same size. It seems like to get the best accuracy they need to set number of time steps to the number of tokens to be generated. Does it make it comparable in speed to an LLM?
Update: finished the paper, and as I suspected, there's a serious downside in speed and memory consumption. LLaDA model has to process the entire output sequence on every time step - without anything like KV cache. Also, full quadratic attention happens on the entire output sequence on every time step, which makes it unfeasible for sequence length longer than a few thousand tokens.
I last used dropbox ~15 years ago. Since then I used S3, Google Drive, Synology NAS, and iCloud. I don’t need to do any syncing across devices: if I need a file I know where to find it (gmail if it’s a document, icloud if it’s an image/video). I still back everything up to Synology.
reply