I've been using Copilot non-stop on every hobby project I have ever since they've let me in (2021/07/13) and I am honestly flabbergasted they think it's worth 10$/mo. My experience using it till this day is the following:
- It's an amazing all-rounder autocomplete for most boilerplate code. Generally anything that someone who's spent 5 minutes reading the code can do, Copilot can do just as well.
- It's terrible if you let it write too much. The biggest problem I've had is not that it doesn't write correctly, it's that it think it knows how and then produce good looking code at a glance but with wrong logic.
- Relying on its outside-code knowledge is also generally a recipe for disaster: e.g. I'm building a Riichi Mahjong engine and while it knows all the terms and how to put a sentence together describing the rules, it absolutely doesn't actually understand how "Chii" melds work
- Due to the licensing concerns I did not use CoPilot at all in work projects and I haven't felt like I was missing that much. A friend of mine also said he wouldn't be allowed to use it.
You can treat it as a pair programming session where you're the observer and write an outline while the AI does all the bulk work (but be wary), but at what point does it become such a better experience to justify 10$/mo? I don't understand if I've been using it wrong or what.
"how to put a sentence together describing the rules, it absolutely doesn't actually understand how "Chii" melds work"
The more experience I get with GPT-3 type technologies, the more I would never let them near my code. It wasn't an intent of the technology per se, but it has proved to be very good at producing superficially appealing output that can stand up not only to a quick scan, but to a moderately deep reading, but still falls apart on a more careful reading. At least when that's in my prose it isn't cheerfully and plausibly charging the wrong customer or cheerfully and plausibly dereferencing a null pointer.
Or to put it another way, it's an uncanny valley type effect. All props and kudos to the technologists who developed it, it's a legitimate step forward in technology, but at the same time it's almost the most dangerous possible iteration of it, where it's good enough to fool a human functioning at anything other than the highest level of attentiveness but not good enough to be correct all the time. See also, the dangers of almost self-driving cars; either be self-driving or don't but don't expect halfway in between to work well.
I wholeheartedly agree with your analysis, but feel like it’s ignoring the elephant in the room: writing code is not the bottleneck in need of optimization. Conceiving the solution is. Any time “saved” through Copilot and it’s ilk is immediately nullified by having to check it’s correctness. From there, the problem is worsened by the Frankensteinesque stitching together of disparate parts that you describe.
I can’t imagine how Copilot would save anything but a negligible amount of effort for someone who is actually thinking about what they’re writing.
What I want is a copilot that finds errors ala spellcheck-esque. Did I miss an early return? For example in the code below
def some_worker
if disabled_via_feature_flag
logger.info("skipping some_worker")
some_potentially_hazardous_method_call()
Right after the logger call I missed a return. A copilot could easily catch this. Invert the relationship. I don't need some boilerplate generator, I need a nitpicker that's smarter than a linter. I'm the smart thinker with a biological brain that is inattentive at times. Why is the computer trying to code and leaving mistake catching to me? It's backwards.
> I need a nitpicker that's smarter than a linter. I'm the smart thinker with a biological brain that is inattentive at times. Why is the computer trying to code and leaving mistake catching to me? It's backwards.
Yes, that's a lot more interesting. Firing a "code review" bot at my code where it asks me questions would be potentially interesting. Even if it sometimes asked some blindingly stupid questions, if I was not obligated to respond to them, I'd be OK with that.
The main problem with that is, GPT-3 can't do that. Personally, while I sing the praises of GPT as a technology, and I do mean it, at the same time... it's actually not a very useful primitive to build further technology on. The question "if you were to continue this text, what would you continue it with?" is hard to build much more than what you see with Copilot. Without a concept of "why are you continuing it with that?" (which, in some sense, the neural net can answer, but the answer exists in a way that humans can not understand and there is no apparent practical way to convert that into something humans can understand).
So GPT-x may yet advance and is fascinating technology, but at the same time, in a lot of ways it's just not that useful.
It reminds me of the video game world, where we have just staggeringly unbelievable graphics technology, and everything else lags behind this spike. Being visual creatures, it causes us to badly overestimate what's actually going on in there. Similarly, it's great that AI has these talkerbots, but they've made a whole lot of progress on something that gives a good appearance, but doesn't necessarily represent the state of the art anywhere else. This AI branch of tech is a huge spike ahead of everything else. But it's not clear to me this technology is anything but a dead end, in the end, because it's just so hard to use it for anything truly useful.
Business is trying to commoditize software development, because it's slow and expensive. All they have ever wanted, is to get their automation with the blanks filled in correctly, more or less immediately.
No-code, visual programming, gherkin, even SQL are all prior attempts at reducing the expense of software development, and of sidestepping the expensive, excuse laden gatekeepers that are software developers.
Copilot is an MVP of a technology that will probably eventually succeed in doing this, and my guess is, it's going to make CRUD slinging obsolete very soon.
Copilot is not backwards, it's just that it's a convenience tool for the execution of business, not for software developers.
When version 2 of the tool can both code and error check, hopefully you're already promoted to architect by then...
What problem does the following pseudocode have?
def some_worker
if disabled_via_feature_flag
logger.info("skipping some_worker")
some_potentially_hazardous_method_call()
And receive this response:
The problem with this pseudocode is that there is no "end" keyword to close off the "if" statement. This means that the code after " some_potentially_hazardous_method_call()" will always be executed, even if the "disabled_via_feature_flag" condition is true.
And that's with a GPT3 without any special fine tuning. Of course, the name `some_potentially_hazardous_method_call` is pretty leading in itself. I rewrote the prompt slightly more realistically, as:
What problem does the following code have?
def optionally_do_work():
if disabling_flag:
logger.info("skipping the work due to flag")
do_work()
and received:
The problem is that the code will still try to do the work, even if the flag is set.
This does seem like a pretty trivial easier-than-fizzbuzz question to be asking, though, since it's so encapsulated.
Not that OP's problem is curly braces per se (they would make the error more obvious, but they wouldn't help with a missing return call), but since we're taling about them...
The insistence of a lot of smart people on using whitespace for logic purposes is THE most baffling thing in the IT space. And I mean that.
Why use some, oh I dont know, CHARACTER, to write down what you mean, why not instead use a NON CHARACTER. Now that's a great idea!
Let's use non characters, so a mix of tabs and spaces (which editors can implement AND display in a number of different ways, even depending on individual configuration!) fucks up shit. Using whitespace is also great because copy/paste is now an error-prone exercise in frustration, which is definitely what we want! Oh and also this will make sure that the peasants are not using our beautiful language IN NAUGHTY WAYS, e.g. you can't really write a deeply nested logic in a single function if it becomes a complete abomination of a mess just after like two or three indentations.
No, but seriously, Python's syntax in regards to whitespace, or any language that uses whitespace for control structures, is hot garbage. I understand that there are preferences, coding standard, etc. and I can tolerate a lot, but this, this is the one hill I'm willing to die on.
The insistence of a lot of smart (?) people on NOT using whitespace for logic purposes is THE most baffling thing in the IT space. And I mean that.
If you look at your typical C-like code, it is already whitespace-oriented, you just manually add the braces as line noise, to make it easier to write a compiler for the language (although even that may not be true). It is like using one character variable names, which - other than the trivial places - makes your code harder to read.
If you want to write deeply nested logic in a function: well, don't. But if you insist, I'm not sure how curly braces help you in this case.
You see I've seen this "argument" a million times and it's always the same: braces are somehow visual noise. And that's it. No more arguments, it's just this one. And it's a very weak one, one that is largely about preferences. On the other hand, I listed 3 examples of indentations actually being in the way of the programmer, ESPECIALLY when working with other people. (Another example: Indentations are notorious for not surviving basic text communications between people, imagine sending Python code over email or god forbid something like Skype. Good luck reconstructing the code on the other side. Because indentations have logic attached to them, you can't just copypaste and let your editor take care of the formatting - like you can in ANY OTHER NORMAL C-LIKE LANGUAGE.)
This is one of those arguments I can see both ways, but I ultimately side on the early return side of things over the else branch side of things.
In my opinion, if the function is done its job, it should return. That's what return is for. As the function grows, the else side of the branch gets longer and longer and it is error prone to leave the first branch of the if statement reliant on it.
> I wholeheartedly agree with your analysis, but feel like it’s ignoring the elephant in the room: writing code is not the bottleneck in need of optimization. Conceiving the solution is.
I dunno about this. I know the received wisdom is that "writing the code isn't the hard part", but I think reality is more like "writing the code is only one of the hard parts". There's an awful lot of badly-written code, or code which is only partly correct, or only correct under some circumstances. The only way to make writing code not one of the hard parts is to specify 100% of the functionality, every corner case, and all test scenarios, before any code is written. And then you still have to verify that it was translated correctly into code, which I think we can all agree is another one of the hard parts!
Conceiving the solution is hard, thinking of edge cases, what-ifs, and failure scenarios is hard, creating effective tests is hard, and writing the actual code understandably and correctly is also hard!
Yes, I think you’ve more precisely articulated what I had in mind. The point stands, though: codepilot does not help with the hard part of the job. It solves a problem that only exists for people who aren’t exercising care.
Writing the code is the hard part mainly inasmuch as it forces you to concretize and clarify previously vague notions. In that sense, it's hard to separate "conceiving the solution" from "writing the code," unless you're perhaps one of those rare geniuses who are simply able to dictate ideas fully formed in their head (I'm thinking of the scene in Amadeus when Salieri examines some clean, uncorrected sheet music and is then shocked to discover that Mozart doesn't make copies and he's holding originals).
> I know the received wisdom is that "writing the code isn't the hard part", but I think reality is more like "writing the code is only one of the hard parts".
Writing the code isn't the bottleneck. And there is no point in optimizing some part of a process that isn't a bottleneck.
Anyway, have you noticed that "understandably and correctly" isn't included on the OP's definition of "writing code"? That's for a reason, and it's the most adequate definition to use on this context.
I also agree that I’d never assume copilot is right when it blurts out code, and that “writing code” is not the hard part — but I’d note three things I found from using copilot pretty intensively over the past year or so:
1. It has shifted some of the code-writing I do from generation to curation.
Most of the time, I have to make some small change to one of the first options I get. Sometimes I don’t. Sometimes I get some cool idiomatic way of doing something that’s still wrong, but inspires me to write something different than I originally planned. All of these are useful outcomes — and unrelated to whether someone is “actually thinking about what they’re writing”.
2. It has changed my tolerance for writing redundant code, for the better.
Like many programmers, I tend to optimize my code for readability first, and then other things later when I have more information. Sometimes, my desire for readability conflicts with my desire for code that avoids redundancy (e.g., “oh but if I put these three cases into an array I can just use a for loop and don’t have to write out as much code” etc. etc.) — and my old bias was avoiding redundancy more often than not. But copilot is really great at generating code that has redundancy, which has often helped me write more readable code in quite a few cases.
3. I refactor code way more now.
In part this is because, given code that already works but is not ideal (e.g., needs to be broken into more functions, or needs extra context, or some critical piece needs to be abstracted), copilot does a fantastic job at rewriting that code to fit new function prototypes or templates. IDEs can help with this task, for a few common types of refactoring, but copilot is way more flexible and I find myself much more willing to rewrite code because of it.
Copilot is not what many people want it to be, in much the same way that Tesla’s Autopilot is not what many people want it to be. But both do have their uses, and in general those uses fall into the category of “I, as human, get to watch and correct some things instead of having to generate all things.” This can be very useful. (FWIW, it takes some time to adapt to this; I teach and mentor a lot and I found myself relying on those skills a ton when working with copilot.)
We shouldn’t discount this usefulness just because these systems don’t also have other usefulness that we also want!
I don't think you've actually used it tbh. It's much quicker to read code than to write it. In addition, 95% of Copilots suggestions are a single line and they're almost always right (and also totally optional).
Here's how it actually works in practice
1. Start a line to do an obvious piece of code that is slightly tedious to write
2. Type 2 characters and Copilot usually guesses what you want based on context. Perhaps this time it's off.
3. No matter, just type another 3 characters or so and Copilot catches up and gives a different suggestion. I just hit "tab" and the line is complete
It really shines in writing boiler plate. I admit that I'm paranoid every time it suggests more than 2 lines so I usually avoid it. But in ~year of using it I've run into Copilot induced headaches twice. Once was in the first week or so of using it. I sweared off of using it for anything more than a line then. Eventually I started to ease up since it was accurate so often and then I learned my second lesson with another mistake. Other than that it's done nothing but save me time. It's also been magnificent for learning new languages. I'll usually look up it's suggestions to understand better but even knowing what to look up is a huge service it provides
I swap between programming languages a lot and copilot saves me a lot of "what's the syntax for for loops in language X again?" style friction, stuff with suggesting correct API usage patterns . It just saves on the friction of writing random scripts.
>I can’t imagine how Copilot would save anything but a negligible amount of effort for someone who is actually thinking about what they’re writing.
since I have a right arm swelled up to twice normal size right now and it hurts to type for more than ten minutes (hopefully ok in a few days) I can imagine an advanced autocomplete being really useful for some disabilities.
the person using co-pilot on all their hobby projects described it as working best as an advanced auto-complete, so I guess you should ask them that.
I figure advanced auto-complete should not produce big blocks of code that are more likely to have logical errors in them, since the grandfather comment here suggested that problems show up when you generate larger blocks of code.
this is consistent with the feedback from many of the users we have talked to as well (transparently I am with Tabnine). Long blocks of code are difficult to digest while short quick ones can be very quick AND easy to validate the logic.
Can Copilot write tests? That way it could test its own code and tweak it until it works.
Of course, one would then ask how to verify tests. I suppose Copilot could write meta-tests - tests that verify other tests. That way it could test its own tests and tweak them until they work.
Of course, one would then ask how to verify meta-tests. I suppose Copilot could write meta-meta-tests - tests that verify meta-tests. That way it could test its own meta-tests and tweak them until they work.
Of course, one would then ask how to verify meta-meta-tests...
I've used Copilot to help with writing verbose unit tests. It can do it as long as you keep an eye over it (basically like an autocomplete), it definitely cannot produce robust test cases on its own though. If you try to do that, it won't take "meta-tests" to figure out they don't look right.
If you have a sufficiently well defined solution to a problem, then you have the code. The next step is just to compile it into something a machine understands. In other words, the code IS the solution, there is no difference between the two.
Only for the most trivial problems. Having seen the same problem implemented both with a spaghetti ball of shit vs something well organized that can be easily read and maintained I’m going to hard disagree on this sentiment.
I have, and it was producing impressive results. However, I was trying to learn Rust and to do so I needed to do the hard yards myself. Moreover, in my brief time using it, it was switching me to a code reviewing frame of mind rather than thinking through the problem.
I thought it might be more useful to me for a language I’m already good at, or one I’m not trying to master but just need to get a task done for.
I had to log back in just to thank you for this link. I've encountered these sites before, and told people about them, but this is just such a perfect chef's kiss example. Sheer perfection.
In my opinion it's probably not worth adding unless the highest-confidence suggestion for a particular completion is significantly lower than average. So it informs you when it's especially uncertain and doesn't have anything better to offer, but that's it.
How has tabnine been responding to the development of Copilot? I've looked into it and the most enticing feature to me is the ability to opt out of it using your own code to train (not even sure if that's a correct interpretation)
Wow, it actually sounds like a great tool for someone who doesn't actually know how to program at all but still managed to get a programming job. Sounds like it could be literally years until they realize you don't know how to program and are using a GPT-3-type completer.
Copilot does a great job of example functions like “function that posts a tweet with the current time”
It falls apart when writing actual code that exists in an app. I’m not convinced even the lowest junior dev could get away with not knowing programming.
I ran into the same problem of "sounds true" when testing the limits of GPT-3 as a general purpose "knowledge engine", used to answer questions about the real world. Either it doesn't understand the difference between truth and fiction, or it doesn't care. The output is 95% truthful and 5% outright fabrication. Even worse, it writes better than most humans, so the fabrications often come out sounding extremely convincing.
I once had it generate an entire interview with an author, which was so realistic I was sure it had encountered it verbatim in the training data. The interview was about one of his books. Turns out such a book didn't even exist, but GPT-3 knew real facts like the name his publisher, the names of employees there etc. and wove them into the story.
The best use I've found for GPT-3 is text summarization, it seems to do very well on that front. I think OpenAI are working on a hyperlinked interface that lets you jump to the original source for each fact in the summary.
After looking at lots of these new models, I've come to the conclusion that they're all basically weighted gibberish generators that produce output not entirely dissimilar to old fashioned hidden markov chains -- just more sophisticated. The corpus is larger and the weighting scheme is much more sophisticated (i.e. you can use prompts), but at the end of the day they're sort of just barfing out nonsense that does a better job at tricking humans into thinking they're doing something smart. These models have really just introduced the concept of differing "qualities" of gibberish on a sliding scale from
random words ---> markov models ---> transformer ---> human writer
Inevitably, users of these kinds of models want them to produce more and more specific output to the point that they really don't want what the models produce and instead are just trying to get a computer to write stuff for them that they want. Eventually all the tuning and filtering and whatnot turns into more work than just producing the output the user wants in the first place.
It's just a room of monkeys banging on typewriters at the end of the day.
> it has proved to be very good at producing superficially appealing output that can stand up not only to a quick scan, but to a moderately deep reading, but still falls apart on a more careful reading
Huh, that’s my experience with human-written texts and journalism in particular.
I don't think they are contradictory at all. Aside from the basic "it has value, just less than $10/month" option, they may also just be interested in the tech and are evaluating it in actual use, etc.
GPT-3 and AI-enhanced code completion has had a ton of hype going on, up to and including claims that software development as a job is at existential risk. Using Copilot non-stop to investigate and understand why there's so much hype and coming away with the opinion that it's not worthy of $10/mo is not contradictory. Would you rather someone who hasn't used a product extensively make value claims?
You overlooked an even bigger contradiction! Namely:
> The biggest problem I've had is not that it doesn't write correctly, it's that it think it knows how and then produce good looking code at a glance but with wrong logic.
I cannot rightly apprehend the kind of confusion of ideas that would provoke such a statement.
EDIT: upon careful rereading, I think I misunderstood. The intended meaning is likely closer to: the problem is less so that codepilot produces incorrect code and more so that its incorrect code appears correct at first glance.
You have my sincerest apologies. I leave this thread intact as a testament to my hair-trigger snark.
I won't rightly describe the confusion your post gives me. Nothing he says seems contradictory. It produces good looking code which upon further inspection has faulty logic.
You would expect that from a program that copies a database of all the examples in the world (or whatever) and then just does an autocomplete without any kind of comprehension of what the problem is that is trying to be solved.
Most people don't make purchasing decisions based on the value they create but rather based on some ingrained assumptions about how expensive software is supposed to be. VSCode and many other complex pieces of software are free, autocomplete is built into my OS, and those subscription consumer software that does have a price usually are priced very low—so relative to those, $10/month feels like a lot (even though I hope that practically anything anyone makes the effort to subscribe to produces at least $10 of value for them).
Some companies seem to be leaning into higher subscription pricing (Superhuman and Motion come to mind) and almost certainly produce far more value than their subscriptions cost if you ask me, but there's definitely a mental barrier to value based pricing to consumers, as well as the fact that with so many companies offering cheap/free software, the market isn't solely determined by value created but rather comparison against other software.
You'd think "very smart" programmers would be equipped to do basic cost/benefit analysis. If it costs $900 and provides $1000 in value, that's $100 more than not buying it.
You'd think "very smart" programmers would be equipped to consider additional factors such as the cost of getting reliant on a tool controlled by a single vendor as well as the effects on other things that they care about. You'd think they also understand that people are not machines and reducing a task by a certain time does not neccessarily mean that they can get that much more work more done overall.
That's not true. The median annual pay for a software engineer in the US is a bit under $100k. (The Bureau of Labor Statistics publishes good data on this.) That means most people doing software development in the US are making under $100k.
Almost anybody doing software development at a well-funded tech company is going to be making over $120k/yr, yes. But it turns out there are lots of other kinds of programming jobs, too.
I've had Copilot enabled since early beta. I think it has saved me ... 15 minutes of typing in total? A few times it has caught on to a repeating pattern and filled a tedious bit of [({}{})] -style javascript correctly.
Most of the things it does for me I could replace with a library of snippets if I could be bothered to set one up.
Not really worth a monthly cost equivalent to, say, Disney+ - which I use tens of hours every month just by myself.
If my employer paid for it, I wouldn't scoff at it, but I'm not paying a cent of my own money for it.
My interpretation is that it's fun to use so they use it a lot but not altogether useful (eta: and/or necessary): they didn't miss it on work projects.
It's an extension... You'd have to go out of your way to disable it. It's also completely optional so it's (almost) only additive. The only harm is sometimes it generates suggestions faster than I can press tab to fix my spacing
It honestly might be the kind of product where it might make sense at a higher price point, with more customer buy-in (and support contracts!) to work around its quirks rather than accept everything it spits out at face value. It's a more common pattern than we're used to seeing, I think.
Interesting to hear your experience. I've been using it for over a year, and I've come to appreciate the (modest) productivity boost that it's given me, to the point that I feel $10 per month is probably worth it.
The completions are often trivial, but they save me from typing them by hand. Sometimes they are trivial yet still wrong so I need to make corrections, wasting some of the gained speed. In total these probably won't save me much time on a day.
However, every couple of days there is one of these cases, where it can do tedious work that really saves time and headaches.
Example:
- After writing a Mapper that converts objects of type A to B, I needed the reverse. Co-Pilot generated it almost perfectly in an instant. This can easily save a minute or two, plus the thinking required.
- For a scraper, I needed to add cookies from my browser into the request object. Basically, I pasted the cookie in a string, and typed `// add cookies`, and it generated the code to split the string, iterate over each cookie value and add it to the correct request field.
So if a few of these cases can save 10 minutes in a month, I feel it's objectively worth it. Then subjectively, not having the headaches of 'dumb stuff'/boilerplate feels great, and I am glad to spend my energy on the actual hard stuff. I will sign up as soon as their sign up page lets me.
I'm the CTO for a small(ish) software consultancy. $10/month is a no-brainer price for just the "amazing all-rounder autocomplete". Spending $10/month/dev to help maximize highly billable engineers? It's well worth the price.
Yeah I was thinking this... Does codex provide other directed sources of learning. I guess I should look deeper into textsynth/Bellard's gpt stuff as it would be nice to have some specialised learning here like only http level interfaces with client server or whatever... But a rails like gen probably gets me there way faster
Of course GPT-3 doesn't "understand" what you are doing. All it's doing is generating high probability text based on a huge training corpus. It's guessing what text will come next. That doesn't mean it understands jack squat. It's basically a parrot with a huge database. Polly want a program?
The fact that such a program is so hyped, is actually an indicator of how much boilerplate and wheel-reinvention goes on among programmers every day.
The state of the sector is somewhat embarrassing. We have armies of monkeys well-paid to bang out the same Java/Javascript/C#/Python over, and over, and over...
Microsoft spent a Billion dollars for an exclusive license of GPT-3, and now they want their return. Expect to see GPT-3 hyped on every platform (including Github).
Github Copilot is on the fence for me between yes/no at $100 per year. I agree that you should rarely if ever allow Copilot to write multiple lines, as your double-checking or debugging time is going to exceed your time savings — the probability of good-looking but bad code is just that high right now. In order to experience a net time-win you'll likely want to be an intermediate at whatever you're doing.
I think "SAAS fatigue" is a thing that needs to be considered. The SAAS model is great for startups and companies seeking recurring revenue. But the modern developer stack now involves dozens of companies gunning for a $5-10/month slice of the pie.
In isolation, most developers could easily afford the $10/month for copilot. But most developers are probably using the free tier for half a dozen services. So the question isn't "Can I afford copilot?", but rather "Does copilot provide more value than upgrading plans on some other service?". For example, if you are using the free tier on Slack, maybe upgrading to the paid tier so you can access the full chat history provides way more value than copilot.
Also, another consideration is that $10 per month is certainly small. But I generally use software I purchase for multiple years. I would guess on average I use a piece of software for 3-5 years. If Copilot was offered for a single purchase price of $300-500, would you pay for it? Because that is likely how much you will spend over the lifetime of the subscription. For me, that price point is approaching the territory of professional tools like CAD software, Photo/video editing software, etc...
I can certainly see why Copilot would be worth $10/month. But I also could see why someone might be uncomfortable with that.
The time savings win is really that marginal. I'm not sure I can save more than 3 hours per year with Copilot. And this isn't saving 3 hours in a single week, this is saving a few seconds here and there accumulated over a year.
Saving time with Copilot is itself a learning process and a probabilistic affair. Copilot can win you a few seconds at a time, but can easily set you back minutes if you aren't careful or experienced. It's the probability of a downward spike in time-win that makes it such a gamble. Such complex deals just turns on the cautious side of my brain.
I'm saying that at $100 yearly I'm on the fence of maybe yes or no. At $60 yearly I'm auto-yes without having to think in rational terms. I guess I'm just not at that place in life where $100 is the tier in which I think emotionally.
Also, if I magically knew that I could save you 3 hours yearly, but it were spread out over the course of a year, and that your savings would occasionally spike down into negative and then slowly climb up, I just wouldn't entertain such a complex offer at such low numbers. People pay insurance just to avoid such incidental downward spikes.
Copilot's biggest limitation right now is that you can't dare to allow minutes of savings per day without inviting the risk of a severe spike in debugging time, the kind that wipes out all your savings. This means you cannot spike up.
- Money is worth more to me than the average US dev because I earn less than US developers, and therefore my time is definitely worth less.
- I cannot use this for work at my current workplace and I'm willing to bet a lot of other companies aren't fine with it either. I'm not saving time where it makes me money, so I would classify as a luxury, not a tool (spending-wise).
Here's how I think about SaaS investments. If it's something I want or am curious about, but doesn't really have a tangible ROI, I decide if it's worth my disposable income and disposable time. If I have neither disposable income nor disposable time, it's not worth it, no matter whether it's $5 or $500/mo. You see, even for $5/yr my time is worth MORE than that money and the cost doesn't make my time worth any more or less.
If it has a tangible ROI, then I figure out how much my time is worth, I figure out how much time or other resource the SaaS app will save and then decide if it's worth the tradeoff. For example, I suck at graphic design, so a monthly $13/mo to Canva is worth it to me to save time, aggravation, and headache, not to mention improved quality of results. I know that I save myself much more in time than the $13/mo is worth.
On the otherhand, I can't justify paying even $15/mo for a podcast transcription tool because I still have to spend dozens of hours checking the transcription and it doesn't save me any headache. So it's not worth it to me. It doesn't matter if it's $60/yr or $100/yr, my time is still worth the same. If it's not worth it at $60/yr , it's not worth it at $100/yr.
Maybe this thought process is different for others, but with so much SaaS out there, it's important to focus on what will drive high value. Incremental "auto-yes" spending at any price point can get you into trouble.
In the long run, I expect very few people will pay for a dedicated Copilot account. It will get bundled in some "development enterprise bundle", heavily discounted. Employees in medium and large shops will just receive it as standard, with their VS or Github paid licenses. Which means it will actually cost half the sticker price.
Licensing is a critical question that is often not considered. Code trained on non-permissive code (think Oracle API's) has very significant risk, ask Google. We took a different tact three years ago in building Tabnine BUT went with only fully permissive code for training, ability to train on your own code base, and zero sharing of your completions. Also we give the developer the flexibility to adjust the length of completions if you want faster shorter suggestions.
It's interesting to draw parallels between the way you describe it and the way more general large language models (LLMs, of which Copilot is in a sense, a specialized instance, applied to code, instead of general language) operate: they also always "know" how to answer any specific question, or how to complete any prompt, without any exception. A model which would be able to "show restraint", and "know when it doesn't know", would be a really impressive improvement to this technology in my opinion.
There are language models that have an internal search engine, they can copy/verify the facts they generate from the source. They are also easier to update, just refresh the search engine. Now you have to provide a collection of "true facts".
As a productivity booster I think it’s worth more than $10.
The licensing problems make it impossible to use at work so I won’t use it for that.
People need to be aware of the security risks of letting microsoft read all your code as it’s sent to the servers copilot runs on. By my lights that’s almost as big of a problem as licensing.
> Due to the licensing concerns I did not use CoPilot at all in work projects
I've rarely found that CoPilot produces more than a line or two of accurate code. How likely is it that one would run into licensing issues with a single line of code that looks similar to something from another codebase?
While I understand the problem in principle, I am really skeptical that significant licensing issues would really come up with using CoPilot as an individual.
I think your response highlights why individual developers are the worst target market and why you want to sell to businesses if you're in the tool space.
Let's say the average developer in the US costs 10k a month (I think that's pretty close to the real average of around 120k a year). So copilot would cost .1% of that developer's salary. I realize calculating things around "improvements in developer productivity" involve lots of fuzzy math, but it would be stupid for any company NOT to pay this if it improves developer productivity by just 1%.
Another way to think about it that I think may be more "real world": Let's say I'm CTO of a big company with 1000 software developers. Do I think it's going to be a better investment to hire another developer so I have 1001 developers, or instead use that other developer's salary to buy all the devs at my company a Copilot license?
But for some reason individual developers think that anything over $1-2 dollars a month is an exhorbitant cost.
If you're an employee what percentage of that productivity increase accrues to you?
I would not spend $10/month on a code completor for my job, because I would probably never see those $10 back in salary. I doubt the company would even notice the minor bump in productivity, say $100 a month
> - It's terrible if you let it write too much. The biggest problem I've had is not that it doesn't write correctly, it's that it think it knows how and then produce good looking code at a glance but with wrong logic.
So the same problem ML has in every endeavor where we have a good metric of "correctness" that's distinct from plausibility, like OCR or natural language translation: very good at spitting out stuff that superficially resembles training data, and whether that happens to be right is totally accidental. Surprisingly good odds if you're working on something boring within the "bounds" of the model, sure, but also pretty likely to think that "now on sale" is a zillion times more likely to be announced on an advert than "a decision has been made to release this product (at an unspecified future date)."
working out what it is worth is tricky, I just look at what I'm paying for at the moment, like Jetbrains All product suite costs me $150 yearly.... so Copilot at $100 a year seems insanely out of proportion. In fact all of the SAAS type products that are around $10 a month per dev offer far more significant functionality. For me copilot mainly fills in boiler plate code, which is useful, from time to time it generates a function that's great, but it would have been trivial for me to write it too, I like it, but comparatively compared to other tools, its pricing seems out of whack. Some are trying to use time = money as justification, but it rarely works as a direct translation like that when coding, in fact I doubt it has that much impact on time, it just makes some things less mundane to do. I have other plugins that are free that probably save me way more time, if all the plugins that help started charging at the same rate based on time savings, it'd be a nightmare.
it thinks it knows how and then produce good looking code at a glance but with wrong logic.
This is so accurate. I still like copilot and I might even pay for it, but I will never trust the logic. It always wrong in a way that _almost_ looks right.
I've been using Copilot for almost 10 months, useful when learning new code but after a while become a bit more advanced auto complete.
I think it is good for short lines, repeating tasks; for example when writing tests and want to assert different fields, assert string, int, etc; for these sort of lines was really good and fast.
my main problems:
1. sometimes make a horrible mistake, takes couple of minutes to understand
2. repeat the same mistake over and over
3. adding a single tab take a bit of time, had to copy & paste tab to avoid copilot suggestion!
I've used it only slightly longer and had pretty much the same experience. I find it hard to believe anyone is really letting copilot write more than 2 lines of code for them at a time.
It's an AI powered autocomplete. And honestly it's excellent at that. All I really want is an AI powered autocomplete and if a FLOSS project took up the challenge I'd happily donate $10/month to see it succeed. Especially if it meant none of the licensing concerns that come with GH Copilot
+1. I've also been using it for hobby projects and have largely the same conclusions. I really do like that it spits out boilerplate for me, but when doing more than that, I still have to double-check all of it because as you said it does create incorrect, good looking output.
I can't justify $10 a month for it. Maybe as it improves.
EDIT: To clarify, $10 a month for personal use. We can't use it at work due to licensing, or it'd be worth that just to emit boilerplate.
I'd be willing to pay more than 10$/month for it personally. It has a greater impact than Netflix on my day-to-day. Also, I would expect my work to expense it.
I turned off copilot a week ago for the same reason. The code it generates _looks_ right but is usually wrong in really difficult to spot ways but things you’d never write yourself.
- It's an amazing all-rounder autocomplete for most boilerplate code. Generally anything that someone who's spent 5 minutes reading the code can do, Copilot can do just as well.
- It's terrible if you let it write too much. The biggest problem I've had is not that it doesn't write correctly, it's that it think it knows how and then produce good looking code at a glance but with wrong logic.
- Relying on its outside-code knowledge is also generally a recipe for disaster: e.g. I'm building a Riichi Mahjong engine and while it knows all the terms and how to put a sentence together describing the rules, it absolutely doesn't actually understand how "Chii" melds work
- Due to the licensing concerns I did not use CoPilot at all in work projects and I haven't felt like I was missing that much. A friend of mine also said he wouldn't be allowed to use it.
You can treat it as a pair programming session where you're the observer and write an outline while the AI does all the bulk work (but be wary), but at what point does it become such a better experience to justify 10$/mo? I don't understand if I've been using it wrong or what.