I just canceled my Copilot subscription last night. I definitely never saw the productivity boost that I’ve seen so many claim.
I actually feel like the suggestions seemed to get worse during my month of using it for some reason.
Towards the end, it started suggesting these large blocks of code (another issue I had with the interface as well) which were very not relevant to what I was attempting to write.
All in all, I’m very underwhelmed from my first experience with “AI enhanced” coding.
Did you change the way you worked to help facilitate it at all?
I get great results from Copilot, but that's because I do a bunch of things to help it work for me.
One example: I'll often paste in a big chunk of text - the class definition for an ORM model for example - then use Copilot to write code that uses that class, then delete the code I pasted in again later.
Or I'll add a few lines of comments describing what I'm about to do, and let Copilot write that code for me.
Did you try the trick where you paste in a bunch of code and then start writing tests for it and Copilot spits out the test code for you?
A few more notes about how I've used Copilot here:
I sometimes prompt Copilot with if statements and var declarations and it actually ends up suggesting missed corner cases and fixing bugs. Definitely, there's an art and form to using it.
I wonder if this quest for the perfect prompt is dumbing me down, though.
I agree. It has made me better at articulating what I want to do, what a function does, and something as granular as what a particular loop hopes to achieve
Just today Copilot saved me close to 1 hour of typing and replaced it essentially with ~1 minute of pressing Tab.
I was hooking IAudioClient COM class to capture and silence arbitrary app's audio, and as soon as I wrote signatures for its members Copilot was able to generate skeletal stub implementation with logging as well as hooking code totaling about 150 lines of Rust.
It would be even better if Copilot was able to make use of the strict typing to prevent generating code that doesn't typecheck in the first place, to be honest. Copilot as-is actually feels more useful for dynamic languages, because static languages get really far without any LLMs and simple static autocomplete gets you really far. So there's more ground to make up.
In addition, Copilot benefits from a large training set, which I guess is best for Python and JavaScript as two very popular languages. Strictly typed languages seem like they are generally less well represented, at least among public GitHub projects.
To offer a counter opinion: while I can live without ChatGPT and cancelled my OpenAI plus membership, I cannot live without Copilot and would be willing to pay a lot to keep it.
There are a lot of tricks to optimizing output from Copilot ("Copilot-Driven Development"), I wrote a little about what I've discovered here:
> Use explicit names, erring on the side of overly-verbose (e.g. ensureUserIsLoggedIn() instead of euil())
I don't care that much if I sound pretentious or too harsh, but how is this not common sense? Maybe the general standards in the industry are way lower than they used to be. This looks like when one of the consultants hired by our company and paid 10k+ per month for a ISO 27001 certification said: you need to make sure your password fields show a * instead of the actual character.
After using copilot for 2 months, I found it somehow useful for very generic repetitive cases, but for complex specific stuff it was very bad. I'd rather spend an hour writing code and understanding the problem I'm trying to solve than spending an hour understanding and adapting some generic code that partially does what it needs to do. And if there are people out there that fully trust it and then deploy code that they don't understand and can't explain themselves, then we're in for a lot of fun in the future.
I've been working as a programmer for almost 15 years and I work on fairly touchy (perf, reliability-wise) networked systems in a programming language with a good type system. I find Copilot to be very valuable even if I have to give it a little bump or edit what it produces sometimes.
I'm definitely not more junior, and I'm seeing massive productivity improvement from Copilot and GPT-4 - but I'm finding it takes a lot of expertise to get the best results, both in my ability as a programmer and in terms of knowing the best ways to use the AI tools.
Learning how to get the best results of them takes a great deal of experimentation.
This has been my experience as well. I actually avoid recommending Copilot to juniors because I think using it requires a deeper understanding of what you're trying to do than is required to just write the code by hand. When you have that understanding it can be a huge time saver, but it's not something you can just pick up and magically ask to solve your problems for you.
I’m curious what your workflow is like switching between copilot and gpt-4. I typically have an open window for each, though this can feel more cumbersome than necessary at times.
I actually use the ChatGPT UI more than copilot - I write code with it while I'm out walking the dog, by describing what I want, then copy and paste it into my text editor on my laptop when I get home.
My experience was very similar. Copilot became a sort of improved autocomplete, that would save copy/pasting repeated variable names into tests, logs, etc, but offered no other significant improvements. Its suggestions were frequently incorrect, often not obviously so.
> that would save copy/pasting repeated variable names into tests, logs, etc
This is all it takes for me to get value out of my subscription. It's saving me time by anticipating my next test / block of code based on what I've previously written, and that saved time really adds up.
Sure it's not "Jesus take the wheel" level AI yet, but time is money, and so is my sanity.
I find it’s really similar to having a team of junior devs under you. Lots of output, generally good for problems that are googleable, need heavy guidance if the problem is both new and niche, lots of subtle bugs.
The first two points in @gavinray’s methodology being unfortunately critical.
This month, for the third time, I evaluated copilot. It has vastly improved. So far that this time I'll probably pay for it.
I think it's biggest benefit isn't for me, though. Copilot is great at churning out predictable or repetitive stuff, much less in writing a abstractions over or around that repetitive or predictable stuff. E.g. it's great at writing yet another CreditcardPayment ActiveRecord model for Rails. But less so for an obscure business domain model used outside of a framework. And great at writing fifteen getters and setters, less for an abstract base class, DSL or macro that introduces all these getters and setters for me.
It's also bad at certain rewrites. A codebase was slowly rewritten from OldStyle (and old libs) to NewStyle. It stubbornly kept suggesting OldStyle snippets.
And last, I find Copilot has a far higher ROI in some languages and frameworks than others. E.g. dynamic languages (like Ruby) have very poor LSP and intellisense support compared to typed and static (like Rust). So the added benefit differs a lot based on where it's used, is my experience.
I guess esp. the latter is why I too am underwhelmed. But also why I'll keep using it, this time, for when I do the inevitable Rails, JavaScript or Panda's gigs. But less so for my weird eventsourced, hexagonal, Rust project.
I also find Copilot to be rather underwhelming and half the time I find myself wishing it didn't go as overboard with the suggestions as it does. Sometimes it suggests exactly what I want to type and then adds a bunch of extra crap towards the end so it is the same amount of effort to type it manually or accept the suggestion and erase what I don't need.
Also its inability to match parens and braces/brackets is legendary. So annoying.
But! It is extremely helpful for filling in error messages. I don't think I wrote a `throw new ` line fully since I started using it. The quality and information density of my error messages has increased considerably.
Not sure if it is worth keeping for just that but it is a nice benefit.
I found the same, and I was really hoping for a productivity boost. I really wonder how much this correlates to developer speed. I code using Vim and type probably 80-100wpm. Typing is usually not the bottleneck for me. Thinking is. Copilot is good at typing, but not good at thinking. IME. Occasionally it will surprise me with a valid contribution, which I then need to check just as carefully as if I were writing it myself.
TBH, using this as a fancy smart autocomplete doesn't seem like the best idea. I can't believe that I'm suggesting something that sounds like Clippy... but...
It really needs to be something where you can rattle off a list of requirements and have it build the code for you. The code context is necessary, but not sufficient.
I don't get the thing with automated PR and commit messages. They mostly describe _what_ was done and not why.
Also, if you enhance your available information with AI, why not just write no message at all (or a short one as always) and use the AI when _looking_ at the commits/PRs?
The AI will certainly be better at this in a later stage because they themselves will get better and they will have more context due to being able to look at the commits that followed the PR/commit.
There is no point in generating the commit message when committing.
When auto-generated messages will become a commodity, there will be issues with it. For example, if the message doesn't fit with the actual commit contents (semantically or syntactically), I have to think hard whether this is an AI-message-generator-bug, or the author missed something or wether I am missing something. This is not cool and makes reading commit messages harder.
Yeah not worth it at the moment. Often it annoys me be overwriting legitimate suggestions from the editor.
Only reason I still have it is because the beta claims to have GPT-4 integration and that could prove to be a very powerful feature. But I still haven;t gotten an invite to install.
Are there other people who find that IntelliJ's autocomplete suggestions and shortcuts (including live templates&Co) already do 80% of what Copilot is trying?
People love to cargo-cult. It's wild. It was okay but I'm happier with ChatGPT for one-off things I don't know how to do "give me a sql query that does this obscure thing" etc..
It's okay that it didn't fit well into your workflow, but accusing those who use it of cargo culting isn't constructive. Other people don't have the same workflow as you do, and as simonw noted above, some of us actually adjusted our workflow to get the most out of Copilot.
I get that gitlab needs more features to continue to sell.
But honestly for me it reduces the attractiveness of gitlab. Just adding half working features does not increase the value for me.
Please finish and polish the existing Features before adding hype feature 93939. Maybe add a policy that your open feature and big count should be less than 20K or something.
Examples:
Security scanning has a god awful UI and UX.
The whole project management stuff is barely usable
Pipelines have loads of weird half features and missing QOL. E
G. Adding saeif support instead of just your own warnings format.
It's honestly doubly annoying when there's multiple times per month where I run into some issue / straight up bug which I then find an ongoing bug ticket from like 2018 for, that's still open but apparently not being worked on.
I really like Gitlab, I think the CI experience is a lot better than, for instance, Github Actions, but I also don't ever see me using a good chunk of the stuff that's been shoveled on in the last few years and I don't really know anyone that does.
At least for python and dotnet, I think github actions are superior.
Gitlab did not support poetry for ages there where bugs with python galore and for dotnet they do not speak the same language - e.g. issues in dotnet are reported as saeif, but gl has it own weird format.
And yes, you are totally right with the bug tracker. It's often funny how long the ticket is just because it was retagged sooo many times.
Crucial: do they train on GPL code? If I am to use this tool, I must abide by the license terms of the training data. Even if it is found that the GPL does not cover LLM responses as derivative works, I would prefer to be on the safe side and refuse to use models trained on software with non-permissive licenses unless I am building Free Software.
Even MIT licensed code requires you to preserve the copyright and permission notice.
If a human did what these language models are doing (output derivative works with the copyright and license stripped), it would be a license violation. When humans want to create a new implementation with clean IP, they have one team study the IP-encumbered code and write a spec, then a different team writes a new implementation according to the spec. LM developers could have similar practices, with separately-trained components that create an auditable intermediate representation and independently create new code based on that representation. The tech isn't up to that task and the LM authors think they're going to get away with laundering what would be plagiarism if a human did it.
Why can't AI do the same: copyrighted code -> spec -> generated code.
... and then execute copyrighted code -> trace resulting values -> tests for new code.
AI could do clean room reimplementation of any code to beef up the training set. It can also make sure the new code is different from the old code at ngram-level, so even by chance it should not look the same.
Would that hold up in court? Is it copyright laundering?
Language models don't understand anything, they just manipulate tokens. It is a much harder task to write a spec (that humans and courts can review if needed to determine is not infringement) and (with a separately trained tool) implement the spec. The tech just isn't ready and it's not clear that language models will ever get there.
What language models could do easily is to obfuscate better so the license violation is harder to prove. That's behavior laundering -- no amount of human obfuscation (e.g., synonym substitution, renaming variables, swapping out control structures) can turn a plagiarized work into one that isn't. If we (via regulators and courts) let the Altmans of the world pull their stunt, they're going to end up with a government-protected monopoly on plagiarism-laundering.
> When humans want to create a new implementation with clean IP, they have one team study the IP-encumbered code and write a spec, then a different team writes a new implementation according to the spec.
Maybe at a FAANG or some other MegaCorp, but most companies around barely have a single dev team at all, or if they're larger barely have one per project.
There’s a clear separation between the training process which looks at code and outputs nothing but weights, and the generation process which takes in weights and prompts and produces code.
The weights are an intermediate representation that contains nothing resembling the original code.
But the original content is frequently recoverable.
You can't just take copyrighted code, base 64 it, sent it to someone, have them decode it, and claim there was no copyright violation.
From my (admittedly vague) understanding copyright law cares about the lineage of data, and I don't see how any reasonable interpretation could consider that the lineage doesn't pass through models.
> But the original content is frequently recoverable.
What if we train the model on paraphrases of the copyrighted code? The model can't reproduce exactly what it has not seen.
Also consider the size ratio - 1TB of code+text ends up into 1GB of model weights. There is no space to "memorize" the training set, it can only learn basic principles and how to combine them to generate code on demand.
The copyright law in principle should only protect expression, not ideas. As long as the model learns the underlying principles without copying the superficial form, it should be ok. That's my 2c
Machine learning neural networks have almost nothing to do with how brains work besides a tenuous mathematical relation that was conceived in the 1950s.
You can say that if you want to nitpick, but there are recent studies showing that neural and brain representations align rather well, to the point that we can predict what someone is seeing from brain waves, or generate the image with stable diffusion.
I think brain to neural net alignment is justified by the fact that both are the result of the same language evolutionary process. We're not all that different from AIs, we just have better tools and environments, and evolutionary adaptation for some tasks.
Language is an evolutionary system, ideas are self replicators, they evolve parallel to humans. We depend on the accumulation of ideas, starting from scratch would be hard even for humans. A human alone with no language resources of any kind would be worse than a primitive.
The real source of intelligence is the language data from which both humans and AIs learn, model architecture is not very important. Two different people, with different neural wiring in the brain, or two different models, like GPT and T5 can learn the same task given the training set. What matters is the training data. It should be credited with the skills we and AIs obtain. Most of us live our whole lives at this level and never come up with an original idea, we're applying language to tasks like GPT.
I think this view is incredibly dangerous to any kind of skills mastery. It has the potential to completely destroy the knowledge economy and eventually degrade AI due to a dearth of training data.
It reminds me of people needing to do a "clean room implementation" without ever seeing similar code. I feel like a human being who read a bunch of code and then wrote something similar without copy/paste or looking at the training data should be protected, and therefore an AI should too.
I mean those consequences are why patent law exists. New technology may require new regulatory frameworks, like we've been doing since railroads. The idea that we could not amend law and that we need to pedantically say "well this isn't illegal now" as an excuse for doing something unethical and harmful to the economy is in my opinion very flawed.
Is it really harmful to the economy, or only to entrenched players? Coding AI should be a benefit to many, like open source is. It opens the source even more, should be a dream come true for the community. It's also good for learning and lowering the entry barrier.
At the same time it does not replace human developers in any application, it might take a long time until we can go on vacation and let AI solve our Jira tickets. Remember the Self Driving task has been under intense research for more than a decade now, and it's still far from L5.
It's a trend that holds in all fields. AI is a tool that stumbles without a human to wield it, it does not replace humans at all. But with each new capability it invites us to launch new products and create jobs. Human empowerment without human replacement is what we want, right?
Has anyone been able to create a prompt that GPT4 replies to with copyrighted content (or content extremely similar to the original content)?
I'm curious how easy or difficult it is to get GPT to spit out content (code or text) that could be considered obvious infringement.
Tempted to give it half of some closed-source or restrictive licensed code to see if it auto-completes the other half in a manner that is obviously recreating the original work.
I don't know about GPT-4 but you could get ChatGPT to spit Carmac's Fast Inverse square root with the comments and all (I can't find the tweet though…)
I can reproduce when prompted all the lyrics to Bohemian Rhapsody, but my doing so isn’t automatically copyright infringement. It would depend on where, when, how, in front of what audience, and to what purpose I was reciting them as to whether it was irrelevant to copyright law, protected under some copyright use case, civilly infringing, or criminally infringing copyright abuse.
The same applies to GPT. It could reproduce Bohemian Rhapsody lyrics in the course of answering questions and there’s no automatic breach of copyright that’s taking place. It’s okay for GPT to know how a well known song goes.
If copilot ‘knows how some code goes’ and is able to complete it, how is that any different?
Most non-GPL licenses require attribution. So, there are two real choices: either we consider that AÍ generated code doesn’t infringe copyright and we are free to use any code: proprietary, GPL, AGPL, or we have to attribute the generated code to all sources, and thus in practical terms your program has to carry a few thousand copies of MIT, BSD, Apache, etc licenses with different copyright headers.
Unless the LLM is cutting and pasting GPL code, I don't see how this differs from an engineer learning to code by referencing open source projects then creating their own project with its own licensing. I'd imagine these models could be tuned or at least a check could be added that no licensed code is being returned.
If someone learns to program by memorising and then re-using blocks of code, they're absolutely doing it wrong – but, also, they should be able to attribute their code.
Most of the techniques I used, I've invented myself (or learnt from the standard documentation). When I use a technique that I haven't invented myself, I look up where it came from. Half the time, my version is actually radically different (and my attribution is mistaken); the other half, I've remembered an inferior version, so I steal the better version and then attribute it appropriately.
That's one way it's different. There are others. Really, though, we should be asking the question “in what way is this the same as humans learning?”, expecting answers that would convince an education specialist.
LLMs don't memorize and reuse. They don't really have a memory of any kind. I think the problem is they're more different from human learning than you think.
Why would you go on HN and say blatantly wrong information
AI is literally 1-1 equivalent to compression. If you don't believe me, you should check out this demo of GPT-2 as a (for awhile SOTA) text compressor.
An LLM's working memory is just its context window, but the LLM also has embedded data in its parameters, which are set during training to minimize loss. This is effectively a memory of the training data, just as much as a digital photo of my face is a memory of the photons reflecting from my face.
> Most of the techniques I used, I've invented myself
This is a very strong statement considering most code is just rearranging existing patterns. Unless you are doing cutting edge academic research, I'm very skeptical of your claim.
This means my Python code is mostly Pythonic. (The rest is kinda idiosyncratic, but I don't often get complaints.) But also:
> considering most code is just rearranging existing patterns.
I am an outspoken critic of "design patterns". They have their place, if you're working with legacy tooling like C++, Java or Rust, but if most of your work is rearranging existing patterns, you have long outgrown your tooling and you need a better programming language. (Or you're copy-paste programming and need to learn your tooling first.)
I am doing academic research, but that's besides the point. So far, I've learned that it's rather hard to be cutting-edge if you don't look at other people's work: you end up re-inventing all the wheels, and any insight you may have brought is lost in the noise.
Imagine a prompt "Photo of person, Shutterstock ID 132456, with blue eyes instead of brown eyes, watermark removed"
If the prompt returns Shutterstock photo #123456 without the watermark (and with the different color eyes) but otherwise a near identical photo, I think most people would agree the output shouldn't be free to use without buying the original photo license from shutterstock.
To a certain extent, we're betting that these models won't accept or reply to prompts that are that specific (e.g. referencing a specific image for sale on shutterstock by its ID number). Or even just providing the photo in the prompt and asking the model to remove the watermark and upscale the image to a higher resolution.
I'm fearful that LLM's will become (or already are?) an easy copyright bypass tool that can be abused, in the example above, to put companies like shutterstock out of business.
This is the sort of problem regulation might help with.
I haven't read OpenAI's TOS, but I'm curious who owns the output of the model and whether OpenAI is transferring copyright/licensing liability onto the user or if OpenAI is representing that output from the model is 100% free to be used in any way the user wants.
I feel like those kinds of issues could be easily solved in court: "What prompt was used to generate this image? Oh, shameless copyright laundering? Rule in favor of the plaintiff. Case Dismissed"
You could say the same of an artist. If you ask an artist to paint a facimile with modifications, they would. That doesn't mean all of their output is "tainted" by copyright just because they learned to paint by referencing existing works.
I guess we'll see what the legal system says, but it's nearly impossible for me to imagine how anyone could ever justify saying a system trained on potentially millions of separate open-source projects can then be said to universally produce derivative works of any specific project it trained on.
On the other hand, if a developer using this tool then goes and tells it "please write me a C library in the style of GNU libc," then yeah, that is skirting a fine line. But just don't do that.
I mean if you ask the tool to produce a specific algorithm by it's very famous name then that's on you. If you ask DALL-E to draw Pikachu you can't be suprised Pikachu when it's a copyright violation.
Doesn't have to be famous. I gave ChatGPT lines 48-67 of drivers/hwmon/sht21.c [1] and asked it to complete the function. Prompt:
Complete the function for an sht21 driver [code omitted]
What it returned was the copyrighted function, comments and all:
static inline int sht21_rh_ticks_to_per_cent_mille(int ticks)
{
ticks &= ~0x0003; /* clear status bits /
/* Formula RH = -6 + 125 * SRH / 2^16 from data sheet 6.1,
* optimized for integer fixed point (3 digits) arithmetic
*/
return ((15625 * ticks) >> 13) - 6000;
}
Notice how it even included a comment referencing a specific datasheet!
This driver is hardly "famous" or even "notable", because those aren't things LLMs understand. The prompt simply contains enough context to be distinctive and the sht21.c is an old, stable driver in each of the many kernel trees included in its training set.
Regurgitation isn't a particularly rare thing with LLMs, most cases just aren't this obvious.
I don't know, it seems like a bit more of a gray area here than with Pikachu. For example, if I said "write me an implementation of fast inverse square root in rust"... and it did... that's certainly not a copyright violation. And if I said "ok, now port this rust code to C..." and it did, then that's certainly not a violation. But then why should I be penalized because the language I want it in happens to be the original language it was written in?
You can't play both sides. If you ask it for something with a copyright and it gives it to you, it has broken the law governing this. Ask it for underage porn and it gives it to you.. it broke the law.
An impressive iteration on this would be a model that would say "here's my solution, and here's what it's inspired by, you may need to update your software's license and/or distribution if you use this solution."
GitLab seems to be a lot more responsive to feedback. It'd be worth asking them if they can publicly document their training set and provide a list of licenses and copyright notices.
"Codey was fine-tuned on a large dataset of high quality, permissively licensed code from external sources" is not sufficient information to be able to provide attribution and licensing information.
Would it be possible to get a complete list of sources and licenses?
Within this context, what is the definition of "permissively licensed"? More to the point, are the terms of licenses of the code in the training set being honored?
There's not actually anything in GPL or any other major FOSS license that prohibits using it to train an AI. The controversy stems from Microsoft refusing to follow the terms of those licenses. If they could just fulfill their obligation to propagate copyright statements and license text there would be no controversy over copilot.
Thus stating that it was trained only on "permissive" licenses doesn't actually answer the question without defining what a permissive license is.
It prevents bad actors like Apple from ripping off people's philanthropic labor, but it also prevents me from ripping off people's labor. It also focuses effort onto the FOSS project.
I like PyQts solution of having GPL or buy a commercial license.
I suppose I still like MIT/Apache style the best. Even if someone rips them off, we didn't lose progress.
Compile already working code, slap my logo on it, spend millions of dollars marketing it with young good looking adults subliminally letting you know that you aren't cool unless you give me money.
Make your logo a trademark so that others can't use it. It doesn't violate GPL because trademarks are not copyright, and make sure your marketing campaign drives home the point that you are the real deal and all others are ripoffs (including the original).
I mean, people manage to sell bottled water to people who have perfectly good and 1000 times cheaper tap water.
GitLab was rather quick to board the AI hype train this year. Fine - they don't want to look like they're behind the curve, have to please investors, and whatever else.
But IMO there are plenty of other places to add real value across the GitLab product with AI/ML features.
Here, it just looks like they saw GitHub do something and felt a need to copy it. But two years late, and worse.
As a longtime GitLab user (and onetime contributor!), I'm a bit disappointed they're spending so much time on this. I think they're just too far behind.
I'll share a few highlights that I am personally excited about
- Explain and help fix security vulnerabilities. From my personal experience, I often find CVEs hard to read, especially when I am not the author of the code to fix. Getting help from AI can reduce entry barriers and make development for efficient. Security is everyone's responsibility these days. This follows the AI assisted feature to explain code in general. "What does this magic loop with memcpy do?" might not stay magic anymore, easing the path to code refactoring, improving performance, and reduce the resource usage footprint.
- Summarize issue comments. Feature proposals or bug analysis can have long comment threads that require reading time. AI will help get the gist and better contribute to what has been discussed.
Additionally, I learned some new ideas at Cloudland last week, regarding product owner requirements list verification, and end-to-end test automation with AI. Need to create feature proposals :-)
> As a longtime GitLab user (and onetime contributor!),
Thanks for contributing. I'd like to invite you to share your ideas about AI features across the platform :)
When you look at the DevOps lifecycle (image in https://about.gitlab.com/gitlab-duo/) from plan/manage to create, verify, secure, package, release, deploy, monitor, govern - where do you see yourself, and where do you spend the most time in?
All the code examples on this page are doubling any brackets ([], {}, ()). How have they managed that? Not a fantastic first impression of its capabilities...
Not only that, but the Golang example is full of errors. Parameter definitions don't have colons between name and type. The map for seen elements is declared as a, but later referenced to as m. The append instruction references uniques, which is undefined. The return statement also references uniques.
I'm going to give them the benefit of the doubt and assume this is a marketing site problem and not a product problem, but either way, not a good look. Reminds me of Google's big Bard announcement including a wrong answer in the image.
I wish them luck. While I'm not a huge fan of copilot coding AIs, I understand how they are going to be a killer feature for many and I worry that it would create a big moat for Github. I hope that Gitlab is able to close that gap because I really like Gitlab.
I've tried it for a few days a month ago and it was noticably worse than copilot, copilot gave much more one shot completions sooner while codeium was line by line on same code.
Way better than the AWS code whisperer or whatever it's called but still not worth switching. Especially since I trust GH way more than some random company (GH already has access to my code anyway)
I should try Codeium then, Copilot's occasional "heap of code" was one of the most annoying parts of using it to me. Why? Because it looks plausible, so I accept it without actually understanding it.
But using it as a better autocomplete is actually quite nice, for example when I have some more repetitive code to write. And a line-by-line AI completer will at least give me the time to read it piecemeal, which I find much easier. Just like I understood way better when my math teacher demonstrated how to solve a problem compared to reading it from the textbook.
I'm already in the flow with copilot so I'm pretty good at expecting when it's useful. Basically I use it as AI copy paste. When I have variations of methods or conditional logic copilot is great at one shot solutions and it's trivial to verify (if it's wrong somehow then I would have done the wrong thing as well - it just saves me a minute of copy/paste and refactoring) copilot was just better.
Signing up for the free trial funnels me to the trial of Gitlab Ultimate. So assuming that you need an Ultimate subscription to use it after the trial, that's the price. Pricing is here https://about.gitlab.com/pricing/
That doesn't answer the question of what the pricing will be. I'm assuming it's not going to be free. Will it be a standalone subscription, or another thing that gets bundled into the main tiers to justify the ever-increasing prices?
Their service calls Google Vertex AI Codey APIs to generate suggestions [1].
It's fine to integrate Off-the-shelf solutions, but you don't have to implement every tech-hype going viral. I fear AI-assistant features are pushing more prescient features further down the backlog.
This looks very interesting, but as an FYI using LLMs to do machine translation is a terrible waste of resources. The example on their landing page has a "translate.py" sample which, albeit handy, is not something I would do beyond basic string translations.
> using LLMs to do machine translation is a terrible waste of resources.
Could you elaborate on that? LLMs seem perfectly suited for language tasks like translation. They don't seem particularly expensive either, especially compared to hiring a person.
There are already machine-translation services trained and created specifically for that purpose. While it's an amazing realization that LLMs can do this and do it pretty well without having to be trained specifically for this one purpose, training something to do literally all text generation tasks is expensive compared to training something specifically to do language to language translation.
For a maybe more obvious example, say that LLMs ever got good enough to do arbitrary precision arithmetic on numbers up to hundreds of digits. Would that be a good use of one when calculators can already do this and are far cheaper to produce? I guess it makes no difference from a free-tier consumer's perspective, but it's still more expensive even if you aren't personally paying the expense.
The quality difference is substantial. I don't care if it's wasteful to use something that has many uses for a supposedly narrow task (although I don't see translation as a particularly narrow task anymore than I see writing as a narrow task). I would gladly waste untold trillions of floating point operations for a 1% increase in translation quality. From my experiments, though, it's much higher than 1% increase in translation quality. And regardless of how wasteful the compute is, it's actually cheaper in terms of dollars. Using GPT-3.5 to translate Korean to English would cost about $11 per million words, based on the average characters per token of the small sample of text I gave it. DeepL (the best translation service I could find) costs $25 per million characters, or for my sample text, about $64 per million words. At $11 per million words I can have GPT-3.5 perform multiple translation passes and use it's own judgment to pick the best translation and STILL save money compared to DeepL.
GPT-4 is a much much better translator than Google Translate and the like. You should absolutely be using GPT for translations especially for distant language pairs that quickly devolve into nonsense with Google Translate, Deepl etc
LLMs are the machine-translation services created specifically for that purpose[0], it just turned out they're very good at many other things!
Your analogy would be like saying why use a computer to multiply numbers if you can calculate them using calculator, which is much cheaper. Sure, but if you already have a computer, no need to use a dedicated calculator as wel.
The original paper[0] that laid the foundation for modern LLMs was demonstrated on machine translation tasks. It's one of the primary use cases these architectures were designed for. What other types of models do you have in mind that outperform them?
It seems I need to have a company to sign up for the trial ?
I have a free gitlab account but can't proceed without entering business information as it seems to be tied to the gitlab premium feature set. Am I doing something wrong ?
What nimrod added a chatbot to this page ? It’s downright unusable on mobile, it just keeps popping up with shit. Even closing it creates two more notifications that use up two thirds of the screen
Going to be interesting. Github had the advantage that they were offering Copilot into an audience of people who just defaulted to them. Gitlab on the other hand is for people who are into license wars, etc. so the AI product is going to be offered to a hostile audience.
I actually feel like the suggestions seemed to get worse during my month of using it for some reason.
Towards the end, it started suggesting these large blocks of code (another issue I had with the interface as well) which were very not relevant to what I was attempting to write.
All in all, I’m very underwhelmed from my first experience with “AI enhanced” coding.