We actually have a full page on the methodology we used. We did over 20 experiments on the best way to digest this dataset and ended up settling on just using llms to review each interview in-depth. This helped us unlock some insights that would only be available if we paid someone to actually sit down and read every single interview. All other nlp approaches we used, like topic modeling, tfidf, and the roaring 2010s wordcloud,s were just too boring to even warrant a write-up.
We tried bringing up as many raw excerpts to light as they make our point that much clearer. People are not simply "happy" about ai as Anthropic would have us believe, they are torn by it. It gives them a deep sense of angst and make them question the authenticity of their very existence.
We have a full page on the methodology we used! Let me know if you’d like access to the dataset we created for this.
The aim was not to be scientific but to flush out some deeper meanings from these interviews that typical nlp techniques struggle with.
Ps: Of course we used llm tools as a writing aid, I’d be willing to bet those “signals” probably come from my own writing though and my appreciation of Tom Wolfe. I’ve been told it can be “sloppy” sometimes.
Now you are just being paranoid, lol. You have me wondering now why am I writing like an llm. (strawberry has 3 rs). Are you a bot? Shoot me an email (that we listed on the page) and let's have a zoom call
I actually think things improved substantially when compared to last year. The latest batch of sota models is incredible (just ask any software engineer about what’s happening to their profession).
It’s only a matter of time until other knowledge workers start getting the asphyxiating “vibe” coding treatment and that drama is what really fascinates me.
People are absolutely torn. It seems that ai usage starts as a clutch, then it becomes an essential tool and finally it takes over the essence of the profession itself.
Not using it feels like a waste of time. There’s a sense of dread that comes from realizing that it’s not useful to “do work” anymore. That in order to thrive now, we need to outsource as much of your thinking to GPT as possible. If your sense of identity comes from “pure” intellectual pursuits, you are gonna have a bad time. The optimists will say “you will be able to do 10x the amount of work”. That might be true, but the nature of the work will be completely different.
Managing a farm is not the same as planting a seed.
There’s a sense of dread that comes from realizing that it’s not useful to “do work” anymore. That in order to thrive now, we need to outsource as much of your thinking to GPT as possible. If your sense of identity comes from “pure” intellectual pursuits, you are gonna have a bad time.
This is 180 degrees from how to think about it.
The more thinking you do as ratio to less toil, the better. The more time to apply your intellect with the better machine execution to back that up, the more profit.
The Renaissance grand masters used ateliers of apprentices and journeymen while the grand masters conceived, directed, critiqued, and integrated their work into commissioned art; at the end signing their name: https://smarthistory.org/workshop-italian-renaissance-art/
This is how to leverage the machine. It's your own atelier in a box. Go be Leonardo.
I definitely understand that this is the rational way of viewing it. Leveraging these tools is an incredible feeling, but the sense of dread is always there in the corner.
You can just feel a deep sense of angst in a lot of these interviews.
In any case, I would rather have them and use them to their full extent than to become obsolete.
Becoming Leonardo it is.
People who have ideas seem to forget that a lot of people do not have ideas!
And even if everyone all of a sudden starts having ideas, only a handful will be useful and successful, and all the other ideas, and people who now have their future riding on them aren’t going to do well.
If everyone was Leonardo, what does the world look like? They can’t all be the greatest polymath of the age
The catch is that many professional environments have evolved values that above a certain quality floor reward quantity over quality. Even more so in the US where pointless torment is "work ethic" and pausing to think something through is "lazy" (see Bill Gate's famous quote about hiring lazy people, or "work smarter, not harder" almost being a rebel motto).
Granted, that's not everywhere. There are absolutely places where you will be recognized for doing amazing work. But I think many feel pressured to use AI to produce high volumes of sub-par work instead of small volumes of great work
> see Bill Gate's famous quote about hiring lazy people
I think this is part of why all this is so contentious. There's been a huge culture shift over the last decade and AI is really just a catalyst to it. We went from managers needing to stop engineers from using too much abstraction and optimizing what doesn't need to be optimized to the engineers themselves attacking abstraction. Just look how people turn Knuth's "premature optimization is the root of evil" went from "get a profiler before you optimize" to "optimization? Are you crazy?"
Fewer and fewer people I know are actually passionate about programming and it's not uncommon to see people be burned out and just want to do their 9-5. And I see a strong correlation with these people embracing AI. It makes sense if you don't care and are just trying to get the job done. I don't think it's surprising things are getting buggier and innovation slowed. We killed the passion and tried to turn it into a mechanical endeavor. It's a negative feedback loop
I think you're looking at different timeframes and different types of bugs.
The last 5 years have been a drastic change for me. For over a decade I've used a Linux desktop and a laptop with various OSes. But in the last 5 years we went from my Linux desktop (Arch of all things!) from having "typical Linux issues" to my macbook having 10x more (and my arch machine being incredibly stable).
Yes, big picture stuff everything is stable and general purpose computers have blurred the line with servers.
BUT there's also a lot of frustrating day to day bugs that did not exist even a year ago. This Apple keyboard bug [0]? Infuriating! I personally hit a similar issue multiple times a day where, despite auto correct being disabled, it will change the word previous to the one I'm typing, even though it was already correct. Worse, pressing backspace delete two words. I'm with this user from years ago... "Am I getting older or is it becoming unbearable?"[1]
Many of the big picture things? Fine. But that doesn't mean I'm not being killed by a million little cuts. That's the problem. They're everywhere and when you complain about any single instance it is easy to brush off. But it isn't a single instance. It is eating hours of my day in 5 sec intervals. That's "buggy as shit"
It's a fundamentally flawed analogy. Leo's apprentices learned and improved. They studied under a master and faced serious repercussions if they bullshitted about their ability or what they had accompolished.
LLM capabilities are tied to their model, and won't improve on their own. You learn the quirks of prompting them, but they have fixed levels of skill. They don't lie, because they don't understand concepts such as truth or deception, but that means they'll spout bullshit and it's up to you to review everything with a skeptical eye.
In this analogy, you aren't the master, you're one part client demanding work, one part the janitor cleaning up after their mistakes.
I'm a professional developer, using SOTA systems, and dealing with them is like bargaining with a fucking empathy vampire. It is emotionally draining.
They trick the reptilian part of your brain that you're dealing with something resembling a human being, but if they were one, they'd be described as a pathological liar and gaslighter. You can't go off on them for it, because they don't give a shit, and you shouldn't go off on them for it, because making a habit of that will make you a spiteful, unpleasant piece of shit for your coworkers to be around.
It's one thing when a machine or a tool doesn't function in the way you intend it to. It's another when this soulless, shameless homunculus does.
That’s an interesting point. I do get pretty tired of the “you are right!” I get the upsides to engagement for a chat bot but for real work it is quite draining.
> just ask any software engineer about what’s happening to their profession
I'm a professional developer, and nothing interesting is happening to the field. The people doing AI coding were already the weakest participants, and have not gained anything from it, except maybe optics.
The thing that's suffocating is the economics. The entire economy has turned its back on actual value in pursuit of silicon valley smoke.
Nothing interesting happening in the field? If you've been paying attention the trend over the last two years has been that the problem space that requires humans to solve has been shrinking. It's continuing to shrink. That's interesting. Significantly interesting.
As an engineer that's lead multiple teams including one at a world leading SaaS company, I don't consider myself one of the weakest participants in the field and neither do my peers generally. I'm long on agents for coding, and have started investing heavily in making our working environment productive not only for humans, but now for agents too.
Things like that are only part of it. You can also also up your agents batting average by finding ways to build guardrails and things that inject the right context at the right time.
Like for instance we have a task runner in our project that provides a central point to do all manner of things like linting, building, testing, local deployment etc. The build, lint and test tasks are shared between local development and CI. The test tasks run the tests, take the TRX files and use a library to parse it to produce a report. So the agent can easily get access to the same info as CI is putting out about test failures. The various different test suites output reports under a consistent folder structure, they also write logs to disk under a consistent folder structure too. On failure the test tasks output a message to look at the detailed test reports and cross-reference that with the logs to debug the issue. Where possible the test reports contain correlation IDs inlined into the report.
With the above system when the agent is working through implementing something and the tests don't pass, it naturally winds up inspecting the test reports, cross referencing that with the logs, and solving the problems at a higher rate than compared to just taking a wild guess at how to run the tests and then do something random.
Getting it to write it's own guardrails by creating Roslyn Analyzers to make the build fail when it deviates from the project architecture and conventions has been another big win.
Tonnes of small things like that start to add up.
Next on my list is getting a debug MCP server, so it can set breakpoints and step through code etc.
That’s fascinating. If you don’t mind me asking, what type of software development do you do? Have you tried any of the latest coding tools? Or even used LLMs as a replacement for stack overflow?
Professionally, I do banking. It's a lot of integration work, sprinkled with a little algorithm every now and then. Lately I've been on capital requirements. The core of that is a system called AxiomSL, which is quite a lot of work for one guy to keep running.
In my spare time I write some algorithmic C, you can check that stuff out on github (https://github.com/DelusionalLogic) if you're curious.
I was an early adoter of LLM's. I used to lurk in the old EleutherAI discord and monitor their progress in reconstructing GPT-2 (I recall it being called GPT-J). I also played around a bunch with image generation. At this point nobody really tried applying them to code. We were just fascinated that it wrote back at all.
I have tried most of the modern models for development. I find then to generate a lot of nonsensical and unexplainable code. I've had no success (in the 30 or so times I've tried) at getting any of the models to debug or develop even small features. They usually get lost in some "best practice" and start looping on that forever. They're also constantly breaking style and violating module boundaries.
If i use them to generate documentation I find it to be surface level and repetitive. It'll make a lot of text about structures that are obvious to me just glancing at the code, but will (obviously) not have any context about the thought process that created that code, which is the only part I care about. I can read the code just fine myself. This is the same problem I find in commit messages generated with AI tools.
For the reversing I also do, I find the models to be too imprecise. It'll take large logical leaps that ruin understanding of the code I'm trying to understand. This is the only place I actually believe a properly trained (not a chatbot) model could actually succeed past the state of the art.
I don't really use stackoverflow either, I don't trust its accuracy, and it's easy to get cargo culted in software. I generally try to find my answers in official documentation, and if I can't get that I'll read the source code. If that's unavailable I'll take a guess, or reverse the thing If it's really important to me.
I would love to be able to say the same but I’m literally the only last person in the company still not using AI to code (if anything for ethics reasons, but I also truly do not need it at all), and I am obviously not the only good dev in the company. The gain is highly debatable (especially in delivery, I do not trust self-reports), however there have been recent reports of morale improvement since using AI, so at least there’s that.
Anthropic released 1,250 interviews about AI at work. Their headline: "predominantly positive sentiments." We ran the same interviews through structured LLM analysis, and the true story is a bit different.
Key findings:
• 85.7% have unresolved tensions (efficiency vs quality, convenience vs skill)
• Creatives struggle MOST yet adopt FASTEST
• Scientists have lowest anxiety despite lowest trust (see ai as a tool, plain and simple)
• 52% of creatives frame AI through "authenticity" (using it makes them feel like a fraud)
I guess all the interview quotes almost feel like those fake reviews on websites. They are all true excerpts, though. And we had a lot of fun creating all of those interactive bits. I get the sense there's a sort of ai content ptsd here, even some of the replies here are being flagged as ai, lol
It's not PTSD, it's that I have no clue what you think of the results of your project here when even your comment on HN introducing it is an infodump that came out of an LLM. I can't tell what you think of the results, if you're skeptical of any of it, or if you think it's a smoking gun, or what. I don't know what parts of it you care more about than others. All I know is what the LLM thinks about the project.
Long term upwork Data Scientist here. It seems he had a bad experience, that could have happened on any other place / platform. There are always horror stories about how platforms deal with conflicts (think PayPal, Stripe, Uber...)
Out of >30 contracts I had, only 1 resulted in a dispute, and it was decided in my favor.
I do agree the whole monitoring thing is absurd and I generally don't take jobs from clients that demand hours to be logged through the monitoring app.
If you want to see how the jungle looks like, please have a go at freelancer.com.
The monitoring app is a hoop I guess. I would hate it too because it's a form of control over the work.
I once had a client who, at the negotiation of the contract, wanted me to work from an "office" (an in-laws quarters) for a specific set of hours each week. His desire for that was because he wanted to make sure he was getting his money's worth; that if I said I was charging him for 10 hours, I was spending 10 hours on the task.
I told him that, along with other stipulations, would qualify me as an employee. He and his fiance/business partner insisted it didn't. That was troubling because his fiance was a lawyer.
As a machine learning freelancer, I can definitely understand the main argument of this article.
When I started applying for contracts I always felt overwhelmed by the competition with math PhDs and other developers with brighter pedigrees.
After closing my eyes to the competition, I started offering high quality communication, clear project specifications and complete honesty in terms of project feasibility, just as I would for any regular software project.
After over 30 successful projects, I can't say I developed an Ai that can beat Starcraft, but I definitely brought a lot of value to all my clients.
If you are passionate about machine learning and really like the grind, just go for it. Work finds a way
I'm one of those PhD types, and I would prefer to work with someone that has your skillset over someone who promises the moon and is solely excited about methodology. Based on your focus, I expect the projects you propose will have a measurable impact to the bottom line. And that is what builds reputation.
Would you mind characterizing briefly, or giving examples of, the successful projects you've taken on as a freelancer?
I've thought before about trying to go that road, but the fact is, for almost every ml project opportunity I've come across, I would have had to say, "well, I can try out some stuff, and it might prove extremely valuable, but just as likely as not my POC will not perform well enough to use in production".
Not only is this the right attitude of what I'd want if I sought your services as a client, but this is heartening to hear as someone who's been thinking about getting more into this. Good on you!
