It seems “Codepilot” is being used as a general name for a coding focused language model. A more literal interpretation of the title suggests you can host Microsoft’s model yourself, which does not seem to be the case (unsurprisingly).
I guess I’m just surprised to see it used this way but “coding focused language model” doesn’t exactly roll off the tongue.
I always assumed it a name for the application of a language model, ie I’m using this language model as a programming copilot. Some models making better copilots than others.
Promising. Combine that with llama or BLOOM, perhaps finetuned on code, and perhaps your own codebase/documentation, and you cold have an interesting solution.
What is wrong with the CodeGen model that they are using?
It is a reasonably large model (up to 16B params) that has already been trained on both natural language and code. I would expect it to underperform larger models, including GPT-3.5 and GPT-4, but this should still be very useful for autocomplete and boilerplate in simpler cases. It is a bit under trained compared to Chinchilla, T5, or LLaMA, but it still performs well.
According to the paper[1], this model is competitive with the largest Codex model that was the basis for Copilot originally.
I haven't ran any of these models yet, I had just assumed CodeGen was less performant for "understanding" prompts. You are right that it's probably enough, especially if fine-tuning is an option.
Now, I wonder: as the code-base grows, how often, and how, should such tuning take place?
I think the GPT-3 thing is wrong, but it is not impossible. You can always train an existing neural net to do new things right? Not sure how to handle different whitespace tokens.
the release today for copilot X says it's using gpt4 "With chat and terminal interfaces, support for pull requests, and early adoption of OpenAI’s GPT-4" ~ https://github.com/features/preview/copilot-x
They only positively say that they are using GPT-4 for the new pull requests feature (and one other feature that I forgot). It’s unclear what model they are using for the main copilot code generation feature. It may be that they are only using GPT-4 for features that require the larger context size.
I wonder if FauxPilot's models (Salesforce Codegen family) can be quantized and run on the CPU. I was able to run the 350M model on my machine but it wasn't able to compete with Copilot in any way. Salesforce claims their model is competitive with OpenAI Codex their github description[1]. Maybe their largest 16B model is, but I haven't been able to try it.
This makes me glad to have chosen an nvidia gpu on linux. You get a whole lot of hidden benefits / QoL improvements that just aren't there with amd and nobody really mentions.
You also have to deal with proprietary drivers. This can be mostly alright until it's not. Look at the hidden costs of dealing with Nvidia for Wayland for instance. CUDA lock-in is like DirectX lock-in. It should be preferable to have something platform agnostic, not just to support the current alternatives, but for new players to be able to join by supporting the common, open APIs as well.
I wonder... is MS likely going to make it harder to swap the URL on the official copilot client in light of this? Will they continue hosting the "competition"?
There’s an alternative client that already exists for neovim, it’s actually better than the official client IMO, and should be easy to change the URL there!
The integration with Copilot doesn’t quite compete with MS, because, well, using the extension requires a functioning subscription. You’d be paying MS, regardless of whether you point it to fauxpilot or copilot.
But there are other extensions that are specifically made for fauxpilot which work well.
This is the first I'm hearing of it. I wonder how many others as well are seeing this via HN the first time? I would imagine MS has been aware of this since more or less inception, however, any product threat isn't really about existence, but how viable and how well known/used it is.
Copilot is a million times better then Tabnine. Tabnine was promising, but totally stopped making any improvements to the model after it got bought out years ago.
I guess I’m just surprised to see it used this way but “coding focused language model” doesn’t exactly roll off the tongue.