The author has implied a false dichotomy: positioning the article as “it does 10x or it does nothing” (my paraphrasing) is disingenuous and hyperbolic. My experience is that on several tasks professional devs, including myself, can get to an answer much faster than pre-LLM. For example, I’ve never had to use SQL frequently enough to become an expert. Prior to LLMs, creating queries beyond the basic would take an hour of Googling and keyboard head banging (or find an expert to help who is invariably doing their own job). Now, the same thing takes 6 minutes. Arguably 10x faster for this task. But since I don’t do this often nor have 40 other examples like this, I’d never claim it makes me 10x more productive. But I DO run into 5 or 6 of this and similar examples a week and several others of smaller magnitude. And that has a meaningful impact on my productivity. I could go on to describe in what ways I can see this productivity improvement but the primary point is that it is not all or nothing. An LLM might make me 20% more productive across my week and that is still a big deal when compared with just not having it.
I don't think the author suggest a 10x-or-bust dichotomy. From the end of the piece:
> I expect LLMs have definitely been useful for writing minor features or for getting the people inexperienced with programming/with a specific library/with a specific codebase get started easier and learn faster. They've been useful for me in those capacities. But it's probably like a 10-30% overall boost, plus flat cost reductions for starting in new domains and for some rare one-off projects like "do a trivial refactor".
That "10-30% overall boost" matches your "20% more productive" pretty well.
I try to capture this sentiment all the time in conversations here in America - we are privileged simply to say, “I feel unsafe” and have it taken seriously when vast swaths of the world would simply reply, “you and everyone else, why are you stating the obvious”
The western world’s calibration on what it means to feel unsafe is so far out of whack that it’s often hard to take seriously.
I’ve led the software development at a design agency to build project and client collaboration tools in a web app over the last seven years. The company founders never set out to intentionally run a design agency with custom built software in house and from the outside looking in, building “project management” from scratch would sound like a terrible idea. Yet we’ve been profitable in every year and grown +30% more than half those years and were acquired last year with decent equity payouts. The acquiring company (300m+ rev / year) is now expanding the engineering team to build SW for their adjacent market.
Myself, my boss (former CEO) and current CEO strongly believe in the power and potential of in house engineering, in particular for spaces that “sound on paper” like solved problems. The efficiency and productivity gains from building exactly what you need is hard to quantify but has been proven in our business as indispensable and, we believe, a significant competitive advantage.
Of course, as others have mentioned, you need the right person to make this happen in your business. I would look for someone with proven startup and product development experience that you could ultimately see managing a small team. None of this is cheap but the ROI long term is likely justified given the issues you’ve outlined.
But I assume you know that so you must be asking, "Is there a situation where somebody can accurately understand the expected outcomes of any particular bet where the odds are against them winning and still come to the conclusion that taking the bet is rational?"
And one answer comes immediately to mind although I'm sure there are several: when I'm willing to lose for the entertainment value of having the chance to win and knowing that short term variance means I may in fact win.
There always has been and always will be a “tax” on those who cannot or will not apply math to their financial decisions. Of course the degree to which we provide opportunities to collect this tax is a societal decision and we’ve almost certainly increased the opportunities for tax collectors in the last ten to fifteen years quite dramatically. And of course, de facto the under educated folks, who also are likely to be poor, will tend to be taxed at a disproportionate rate. Yet another wealth transfer. I wonder what societal attitudes led us down this road?
“Historically, many studies have involved the easiest people to recruit, who tend to be people who come from advantaged environments. If we don’t make efforts to recruit diverse pools of participants, we almost always end up with children and adults who come from high-income, high-education environments,” Gabrieli says. “Until recently, we did not realize that principles of brain development vary in relation to the environment in which one grows up, and there was very little evidence about the influence of SES.”
Without this very type of study one could mistakenly do exactly what you’re accusing them of doing. I also didn’t see race mentioned. I’d assume the opposite of your presumption i.e. that this finding would stand regardless of race.
>They were told that for each correct guess, they would earn an extra dollar, and for each incorrect guess, they would lose 50 cents.
Its been proven time and again that these meaningless level rewards are pointless in studies and often even distort/bias the data (people treat it like a board game). Probably only second to people not considering SES even when they are studing SES.
I am more interested in reading about debunked studies rather than new ones, as you can tell I have a huge general assumption that they will be poorly performed.
I guess the one thing that was also left out from my comment was also that most of these experiments are carried out by college age undergrads that are rich (and still mostly white).
Yeah they are "overseen" by someone with a PHD.... who depends on getting as many papers as possible published. Which is another issue. As I said I obviously have some serious bias against the validity of most "social science"
> Its been proven time and again that these meaningless level rewards are pointless in studies and often even distort/bias the data (people treat it like a board game).
Do you have one or two references for this I could take a look at?
I'm as wary of bias and mistakes in research as "the next person" but if they are measuring brain activity it gets more interesting. To establish that there isn't some validity to this experiment then one would have to show that the brain's reward center can respond while receiving a reward while playing a game but not respond when receiving a reward in a different "non-game" context.
If such research exists or it's been established as fact that this is true, that would seem like a glaring omission from this study. I'd be interested to see that.
Secondarily, you avoided addressing the race question entirely :) I'll assume it is safe to say there is some bias there as well then.
That is probably correct. I wasn't specifically dunking on this study about the race component. Its just a huge issue with this field in general. Also(My take) the race doesn't have so much to do with it as the typical culture from what we classify as race but since culture type is not ever tracked anywhere "race" is all we have to go on. There by my take could be someone that is "black/asian/indian/whatever" but raised in "white" culture and would count from a psychology experiment standpoint as white.
Sorry I don't have them, too much cruft online to find them. Mostly as debunks to other experiments where when real stakes are used the experiment could not be replicated. You may be able to find them searching "replication crisis" as one of your keywords but I don't have time to dig them up right now. its just a fringe minor interest of mine.
On a side note: Have you ever participated in any of these? I have and I've seen participants actively undermine the spirit of the experiment while in the experiment, making side bets and such. Usually there is some sort of cap on what you can get 5-20$ or something and people there are basically saying the few bucks was just for beer money or whatever so they are willing to risk it to make a "Pot" out of the money and one person can have a nice day from it. Again since most of the people available for these experiments (despite what TV portrays) are not really hard up for the few bucks from the couple hours they spend participating. (based on my real world experiences)
Could that have been part of the experiments? Maybe. Maybe for some of them, but not all the ones I've seen for sure. And they used that data to "prove" something...
There is one measure of a government that has never failed: are people trying to get in or are people trying to get out?
California's government - federal electives, state and many local have been making decisions and putting in place policies that we are now seeing the consequences of.
The pandemic didn't create the problem - it poured gas on the fire. The elected officials of the last 30-40 years created the problem. The housing shortage is 100% a government created problem. The homeless "problem" is 100% a government created problem.
We don't have to solve these problems - we can continue on as we are which is likely because it is the path of least resistance (accountability is hard for people and governments). But let's be honest about why things are the way they are.
>There is one measure of a government that has never failed: are people trying to get in or are people trying to get out?
Can you provide the units in which this "measurement" is made? What about other measures of government? How does California perform on those scales?
Should it be the overriding goal of government to appeal (or appease) those citizens who are already on the margin of leaving anyway? Or should it strive to better serve those who stay?
