Thing is, and LLM doesn't need motivation or self-discipline to start writing, which at this point I'm confident is the main slowing down factor in software development, after requirements etc.
These also have larger memory in a way, or deeper stacks of facts. They seems to be able to explore way more sources rapidly and thus emit a solution with more knowledge. As a human I will explore less before trying to solve a problem, and only if that fails I will dig deeper.
But they fail at global context, consistency, and deep understanding which constantly fails them in the real world.
You have to basically tell them all the patterns they need to follow and give them lots of hints to do anything decent, otherwise they invent new helpers that already exist in the codebase, don't follow existing patterns, put code in places that aren't consistent.
They are great at quickly researching a lot, but they start from 0 each time. Then they constantly "cheat" when they can't solve a problem immediately, stuff like casting to "any", skipping tests, deciding "it's ok if this doesn't work" etc.
a few things that would make them much better:
- an ongoing "specific codebase model" that significantly improved ability to remember things across the current codebase / patterns / where/why
- a lot more RL to teach them how to investigate things more deeply and use browsers/debuggers/one-off scripts to actually figure out things before "assuming" some path is right or ok
- much better recall of past conversations dynamically for future work
- much cheaper operating costs, it's clear a big part of why they "cheat" often is because they are told to minimize token costs, it's clear if their internal prompts said "don't be afraid to spin off sub-tasks and dig extremely deep / spend lots of tokens to validate assumptions" they would do a lot better
90% of writing code, sure. But most professionnel programmers write code maybe 20% of the time. A lot of the time is spent clarifying requirements and similar stuff.
The more I hear about other developers' work, the more varied it seems. I've had a few different roles, from one programmer in a huge org to lead programmer in a small team, with a few stints of technical expert in-between. For each the kind of work I do most has varied a lot, but it's never been mostly about "clarifying requirements". As a grunt worker I mostly just wrote and tested code. As a lead I spent most time mentoring, reviewing code, or in meetings. These days I spend most of my time debugging issues and staring at graphics debugger captures.
Clarifying either business or technical requirements for newer or junior hires.
> reviewing code
See mentoring.
> or in meetings
So clarifying requirements from/for other teams, including scope, purely financial or technical concerns, etc.
Rephrase "clarifying requirements" to "human oriented aspects of software engineering".
Plus, based on the graphics debugger part of your comment, you're a game developer (or at least adjacent). That's a different world. Most software developers are line of business developers (pharmaceutical, healthcare, automotive, etc) or generalists in big tech companies that have to navigate very complex social environments. In both places, developers that are just heads down in code tend not to do well long term.
The irony is of course that humans in general and software professionals in particular (myself definitely included) notoriously struggle with communication, whereas RLHF is literally optimizing LLMs for clear communication. Why wouldn't you expect an AI that's both a superhuman coder and a superhuman communicator to be decent at translating between human requirements and code?
> Why wouldn't you expect an AI that's both a superhuman coder and a superhuman communicator to be decent at translating between human requirements and code?
At this point LLMs are a superhuman nothing, except in terms of volume, which is a standard computer thing ("To err is human, but to really foul things up you need a computer" - a quote from 60 years ago).
LLMs are fast, reasonably flexible, but at the moment they don't really raise the ceiling in terms of quality, which is what I would define as "superhuman".
They are comparatively cheaper than humans and volume matters ("quantity has a quality all its own" - speaking of quotes). But I'm fairly sure that superhuman to most people means "Superman", not 1 trillion ants :-)
I wrote that based on my experience comparing my prose writing and code to what I can get from ChatGPT or Claude Code, which I feel are on average significantly higher quality than what I can do on a single pass. The quality still improves when I critique its output and iterate with it, but from what I tried, the quality of the result of it doing the work and me critiquing it is better (and definitely faster) than what I get when I try to do it myself and have it critique my approach.
But maybe it's just because I personally am not as good as others, so let me try to offer some examples of tasks where the quality of AI output is empirically better than the human baseline:
1. Chess (and other games) - Stockfish has an ELO of 3644[0], compared to Magnus Carlsen at 2882
2. Natural Language understanding - AIs surpassed the human expert baseline on SuperGlue a while ago [1]
3. General image classification - On Imagenet top-5, facebook's convnext is at 98.55 [2], while humans are at about 94.9% [3]. Humans are still better at poor lighting conditions, but with additional training data, AIs are catching up quickly.
4. Cancer diagnosis - on lymph-node whole slide images, the best human pathologist in the study got an AUC of 0.884, while the best AI classifier was at 0.994 [4]
5. Competition math - AI is at the level of the best competitors, achieving gold level at the IMO this year [5]. It's not clearly superhuman yet, but I expect it will be very soon.
6. Competition coding - Here too AI is head to head with the best competitors, successfully solving all problems at this year's ICPC [6]. Similarly, at the AtCoder World Tour Finals 2025 Heuristic contest, only one human managed to beat the OpenAI submission [7].
So summing this up, I'll say that even if AI isn't better at all of these tasks than the best prepared humans, it's extremely unlikely that I'll get one of those humans to do tasks for me. So while AI is still very flawed, I already quite often prefer to rely on it rather to delegate to another human, and this is as bad as it ever will be.
P.S.
While not a benchmark, there's a small study from last year that looked at the quality of AI-generated code documentation in comparison to the actual human-written documentation in a variety of code bases and found "results indicate that all LLMs (except StarChat) consistently outperform the original documentation generated by humans." [8]
Brother, you are not going to convince people who dedicated their lives to learning a language, knowledge that bankrolls a pretty cushy life, that that language is likely to soon be readily accessible to everyone with access to a machine translator.
Any chance the business/product folks will be using LLMs on their side to help with "clarifying requirements" before they turn them over to the developers?
They view this task as tedious minutia which is the sort of thing LLMs like to churn out.
I received counterfeit goods multiple times due to this. I set up a subscribe and save order and they would let random retailers fill the order with fake products. Amazon collected the money and just did not care, they need to be held accountable for these things.
> they would let random retailers fill the order with fake products
What made this all particularly insidious is that Amazon not only commingled inventory, but actively refused to track where inventory came from.
This meant you only needed one fraudulent seller to poison the entire inventory pool and there was no way know where the bad product came from because Amazon actively avoided being able to track it.
That's the aspect of it that always felt particularly malicious to me.
Amazon don't check returns either. It's a nightmare if you use their FOB service. We've had product returned, not checked and then shipped to another customer who then pputs in a claim because they didn't get what they ordered - because Amazon didn't check the return. Amazon then claim you're selling counterfeit goods.
Entirely why we no longer use their service and ship direct for amazon orders. Some people still try the trick but we always put a claim in and amazon after they automatically give a refund to the buyer, and Amazon pay it. So Amazon pay twice. Maybe the cost of just accepting that loss is less than having someone check the return.
The bad part here is letting “poisoned” inventory in.
Adding vendor tracking adds a layer of ERP difficulty that isn’t practical for bulk, cheap items.
You either have to have serial numbers (unique per item, not just a product identifier barcode) or you have to physically segregate inventory by vendor, which is not practical.
If the vendor doesn’t serialize the item, then Amazon has to add it on receipt. Certainly not worth it for $10-20 item.
Russia has a working system that tracks retail sales of individual cans of beer, bottles of milk and such. Initially it was introduced to track things like shoes and furs that were massively counterfeited, but then expanded to include other goods. So now in a grocery store you use it, for example, for all milk products (milk, cheese, ice cream, etc.), vegetable oil, beer, mineral water. Technically you just scan a different barcode (QR code). There's also an app you can use to scan the thing and get more information, such as the exact producer. The general idea was to fight counterfeit goods, but as a side effect it also enforces shelf life rules or may help to find a drugstore that has a specific drug.
So it is possible and not that expensive even as a country-wide system for goods that cost around $1 (a typical can of beer).
And yes, it does have additional codes for larger-scale packages. So a pack of cigarettes gets its own code, a carton gets its own code, a box of cartons gets its own code. A wholesaler can just scan the box and the system updates the status of every pack inside.
What am I missing about this? Couldn't the scammer just replicate the QR code of a legit shop? I thought the point of counterfeit goods was to fool you into buying them instead of the real thing. I guess part of the process would have to be verifying that every shipment of goods received was accurately tracked from a valid "ship from" address, but I have to imagine there's a lot of common warehousing in use for bulk goods. I'm not understanding how the QR code helps solve that.
Maybe a unique bar code per-item that includes some private hash information that makes it unique to the producer? Sort of an electronic signature for physical goods? Then if there's a centralized database, copying the QR codes wouldn't do much good. You might be able to slip in one if it is sold before the real version. But each subsequent copy could be caught.
This is fascinating in the context how they use and abuse intermediaries to buy and smuggle western tech into Russia. If every chip were that well tracked, it would be a lot easier to clamp down on it..
They didn’t need to actually track things internally, add a sticker or even have someone stamp the vender code to the item listing the vendor when you’re adding the item to the bins and if the customer complains you can likely use that sticker to track who added the item after the fact. Critically you don’t need some 6 digit number for vender code, every new vender for a given item gets a number for that item, software can remember the relevant mapping.
If some vender is adding fraudulent items to the system based on some thresholds you set, charge the vendor to manually sort those specific products out.
Odds are they would make up the ~5 cents per item just dealing with less fraud. However, you don’t need to track every item rack the first few thousand items from a vender and you can scale back tracking as they prove themselves. At scale this could be almost arbitrarily cheap.
There’s many illegal things which can boost a companies bottom line. Quite often the law cares about what’s a reasonable effort which is a very different standard than what maximizes profits.
Something which may or may not decrease their bottom line but definitely significantly reduces counterfeit items ending up in customers hands is going to be considered reasonable even if it’s not profit maximizing.
That’s a really clever and simple plan but doing anything like applying stickers, correctly, by hand or robot, can add cost ranging from $<surprising> to $<shocking>.
Maybe they have a variation of your idea where they inkjet a serial number onto a conveyor belt of incoming items or add a super-cheap chip of some kind.
My understanding is every individual item is tracked in an Amazon warehouse - so Amazon knows that the 67th item in a box from supplier X was shipped to user Y.
They don't just track quantities of SKU's like most other retailers.
This always confused me. You have a bottle of glue sold by company X. Then you have 87 different people "buying" the glue in bulk, having it sent to Amazon, and selling it on Amazon as if it comes from their store:
Buying option 1: Company X glue from store A.
Buying option 2: Company X glue from store B.
Buying option 3: Company X glue from store C.
...etc.
But then Amazon says, "actually, these are all the exact same bottles of glue, so we'll thrown them all into the same bin, and no matter what "store" the people buy them from, we'll just grab them out and send them to the customer.
Now even without counterfeits, this is weird. What exactly is the point of store A, B, C, etc.? Company X sends the bottles to Amazon, they get put in one big pile, you buy them on Amazon, and Amazon takes them out of that one big pile and sends them to you.
The only thing purpose of the "stores" when you co-mingle inventory seems to be:
1. Plausible deniability for counterfeits. Hey, they told us they bought it from company X, we had no way of knowing they didn't.
2. Getting money from people trying to get rich quick in the marketplace. Some people will try all sorts of cuts to boost their Amazon sales in the hope that it will pay off later.
The reality is more complicated than you are assuming. A shockingly large number of vendors grossly mismanage their supply chains such that Company X can actually be legitimately undercut by reseller Company A on Amazon even though Company X produces the product! The mechanics of it are convoluted but legit, and there is a huge ecosystem of companies that arbitrage the legions of producers that are bad at managing their global supply chains.
Amazon has an interest in allowing these resellers of legitimate products to exist because it pushes down prices from the primary vendors, lowering prices for the customer. The primary vendors end up competing against themselves indirectly but they have no one to blame but themselves. This is the milieu in which counterfeit products exist.
If the producers of these products were consistently competent at managing their supply chains it would be much less of an issue because it would clear the field of resellers arbitraging the mismanagement, leaving only Company X and the counterfeiters which is a much easier problem to solve because you don’t have to worry about banning legitimate resellers. But that isn’t where we are.
That’s one way but far from the only one. Producers like to do things like make random deals through their myriad divisions to offload inventory to a random reseller very cheaply that ultimately finds its way onto Amazon at a price that undercuts cost of the original producer’s contract on Amazon. The cost of sales are not the same on Amazon even if you are selling the same product, so they can legit undercut you. You also have different divisions of the same company around the world all selling on Amazon under different contracts competing with each other (which Amazon tacitly encourages AFAICT).
Smart companies put contracts globally that have Amazon implications under a single person who can see across every deal. If they sell to someone with a restriction on Amazon resale, they will mark those goods so that they can track it if it shows up on Amazon. However, there are so many fly-by-night resellers that this is a losing proposition, so many don’t bother with those resellers anymore because enforcement yields nothing.
The vast majority of companies are naive and not very smart about any of this. People that know how to systematically set up a sales program that is profitable and resistant to arbitrage on Amazon get paid a lot of money in industry. It isn’t that hard but most companies can’t seem to figure it out.
That's one approach. There is also "buy it at a discount, resell it without much markup" and "buy it earlier, store it until prices rise", and plenty of other ways to perform this arbitrage.
I read it as "For items in the ten to twenty dollar range, its not worth adding a vendor label" and I don't suppose its the cost of the sticker, but how much longer does it take the warehouse worker to take it from a shelf and put it in a box if they add a sticker to every item? +5% ? +10% ? +100% ? (It takes very little time to put an item in a box, I could see adding a sticker doubling this...)
Is that real? I find it hard to believe that Amazon effectively accepted stock from third parties "as is" and lost track of where it came from. It's more likely that they don't tell you than they don't track.
In the seller documentation they say they can track the source of commingled inventory - they achieve this by never putting them on the same physical shelf location.
A fair point and important distinction, but so is the difference between "we CAN" and "we WILL/DO". That "myth" didn't come out of thin air. It's a result of of amazon not doing that unless they felt it financially prudent to do so/until enough people bitched about it.
The OP article is exhibit A for how common of an issue this was.
We know that they would not provide such tracking for those conducting fraud investigations. You can believe they intentionally didn't track the source or that they intentionally refused to share the information to root out fraud; either one is a very bad look for Amazon.
It wouldn't surprise me. Amazon knows where every item is in its FC and knows the motions of every item's placement, from the grand scheme of things.
It's not that hard to then track back from an order exactly what bin or tote or shelf the item was pulled from, then look at what shipment(s) that bin's items came from to figure out what supplier it came from.
They know the counterfeit goods came in and were stowed to bin XYZ and they know that someone pulled from XYZ...
Its co-mingled, so you may not your stuff back, you just get similar stuff back. That is if you pay to get it back. It’s usually cheaper to let them flog it off cheap on prime day
I don't think ending commingling will stop that from happening, since Subscribe & Save is set to switch to a different seller with a lower price by default.
In the US, when Subscribe & Save is set up, it is set by default to receive orders from "Amazon.com and other top rated sellers". If you want to change it, you need to go into the Subscribe & Save page and change it to "Amazon.com only".
I've had an order where I initially placed a new subscription sold by Amazon.com, but a 3rd party seller would lower their price by a few cents, and Amazon would change the seller and I would receive grey market goods.
I haven't found a way to change the default for new subscriptions to always use the same seller that I set up the subscription with, so I need to manually change it for every single new subscription.
Yeah. This is a joke. They give us a 5-10% discount to do this. But when the time for the next delivery came, they had doubled the prices instead of locking in the price I had subscribed at. I had to cancel the order.
If I had been informed during subscription that fulfillment will be done at the price prevailing at that time, I would never have subscribed in the first place.
Instead of Subscribe and Save, which is almost useless for the reason you give, I wish I could place a standing limit order to buy up to X of something every Y months, but only if it's price Z or better.
Yep. The number of times I've received garbage while paying full price is wild. I finally asked the rep what I need to be looking for before purchasing to avoid this. They didn't really have an answer.
I did get a refund most of the time. Amazon's service is still quite good even today. Already don't feel great about ordering from Amazon but this really made me cut back over the last year or two.
I received counterfeit toothpaste from Amazon during COVID. I got asthma from using it and never had asthma before or symptoms like that. I purchased tubes from Costco to compare and the Amazon ones were clearly fake.
I left a review to warn others on the page. Amazon removed my review and lifetime banned me from leaving reviews again citing “abnormalities.
I've been ordering on Amazon in Germany for a good 20 years now and I've never received a counterfeit item. Is it not a thing here? Does it primarily affect certain countries? Am I insanely lucky?
Been ordering since January 2000 (> 1,000 orders) from Amazon Germany and never received anything counterfeit as far as I can tell.
I think I'm pretty good at spotting fakes, because I'm sensitive to tiny typographical or material-wise quirks. In the same period, I've received multiple fakes on eBay, including a genuine phone that came with a counterfeit charger.
I can imagine that commingling introduces a very low-percentage risk of receiving a counterfeit product but due to the immense scale, it still affects a huge absolute number of orders.
I haven’t it happen to me either (Spain) but if you sort comments by negative you see it happen relatively frequently, and it’s credible reviews with pictures.
I also haven't received any counterfeit ever. I have 400+ orders a year...
I think what a lot of people qualify of counterfeit (not saying OP does here) is people buying cheap no-brand Chinese garbage and receiving cheap no-brand Chinese garbage and not being happy with it.
There’s definitely counterfeit stuff, but it’s much more common in some categories than others. USB cables and chargers, SD cards, USB flash drives for example are commonly counterfeited (e.g advertised as made by Apple or SanDisk but actually being ‘cheap no brand Chinese garbage’). Many wouldn’t even realize it because the products usually still work, but have major defects like less capacity than advertised or being built much worse.
Or buying, for example, electric toothbrush heads at half the cost they are in the supermarket and then being surprised when they turn out to be counterfeit.
But that's the other aspect to commingling, I guess - you might pay full price for the real deal and get a fake, due to bad actors' stock being commingled, but on the flipside a punter who's paid an unrealistically low obviously-fake price might actually get the real deal, adding an air of legitimacy to the bad actor.
No, I know loads of people having bought SD cards from reputable brands at full price, only to receive a fake one in the mail. You don't notice until you're using the card and your gopro writes too fast to it. I don't like the victim blaming of your post.
Amazon regularly commingling legitimate and counterfeit goods, means that customers are left with the job of trying to verify that the goods they ordered are legitimate. For every customer that complaints & refunds, there might be three or more who don't.
Some of these counterfeit products have legitimate safety concerns, for example lead paint usage, battery fire risks, PPE that misstates its effectiveness, or USB chargers with poor AC DC electrical isolation.
This is a huge trust problem, and "the customer needs to detect counterfeits and refund," isn't actually a solution to THAT problem.
Or... Amazon could say "this is a third party seller on our platform, so blame THEM",then the third party seller gets held responsible for the fraudulent goods despite having no ability to control it. Not a very sane system but apparently the one we chose to use (at least until now, when Amazon decides to eliminate commingling).
I bought an electronic item brand new, sold by Amazon, and they sent me a used one that already had its digital bundle redeemed by someone else and 5 out of 12 manufacturer warranty months used. I contacted Amazon support about this within a week and they told me replacement is not possible in my situation, I can return it but a full refund is not guaranteed
That seems like illegal fraud on the part of Amazon. They sold something they claimed was new and it wasn't. They have to give you a full refund if you return it.
And what is their recourse if Amazon just says "nope, deal with it"?
Sue them? Even assuming that kind of time, money, and energy expenditure is within their resources, Amazon's legal department is likely to be able to stonewall until they run out of money.
Then multiply that by however many thousands or tens of thousands of customers Amazon has done this to. No more than a fraction will ever complain, no more than a fraction of those will ever sue.
Amazon has been allowed to get too big, and the usual methods of dealing with fraud at this level simply don't work reliably.
I tried, but they basically said we can't help anymore in this matter and disconnected me and I saw no further way to escalate. This was in the US. In the EU when I had another problem with Amazon and CS was unhelpful I was able to escalate via relevant authorities and it eventually got someone from Executive Customer Relations to send an actual human-written email apologizing for the whole ordeal and resolving my situation
I ordered a quantity of o’reilly books from Amazon because that was the only outlet offered by them.
I received some real books, and several counterfeit copies. The same books weren’t even the same size, some also had thin pages, some yellow pages, and several with printing errors the others didn’t have.
Sadly I tried to contact the publisher and let them know about the counterfeit books in their listings and tried to warn them about what was going on, but their support people only wanted me to take it up with Amazon, and couldn’t understand how to escalate my concerns internally and just kept asking me if they could close the ticket.
Also Amazon refunds aren’t as smooth when you don’t live in the US and already paid customs duties on the counterfeit products, and the return shipping costs make returns prohibitively expensive.
I wouldn’t have even ordered from them in the first place if I could have avoided it.
I’m glad they are solving this problem, but I also kind of don’t want them to succeed because of their terrible legacy.
They are very quick to refund/replace and I wonder if it’s because they have no way to track an item’s origin to a specific seller so they just treat the refund as a small cost in an otherwise very profitable system for them.
They are quick to refund/replace because most of the items sold are returnable and "replace" is just a second order. Basically you have the ability to self service this. So it's a quick way for a CSR to satisfy a customer without doing anything special.
mIRC scripting is what got me into programming. The summer I found it as a kid I was immediately hooked, being able to write code that actually interacted with the world and did something blew my mind. I was so into the 'war channels' where everyone would kick each other and optimization was key.
Getting the dockerfile order right is critical due to how docker caching works.
Even if you aren't an expert it is trivial these days to copy/paste it into chatGPT and ask it to optimize or suggest improvements to the dockerfile, it will then explain it to you.
Just as the engine replaced physical strength, artificial intelligence, through models like large language models, is now replacing cognitive labor and thought.
From the article "Muscles grow by lifting weights" yet we do that now as a hobby and not as a critical job. I'm not sure I want to live in a world where thinking is a gym like activity, however if you go back 200 years it would probably be difficult to explain the situation today to someone living in a world where most people are doing physical labor or using animals to do it.
Artificial intelligence doesn't negate the need for human intelligence.
The jackhammer replaced the hammer and chisel for busting concrete, and the user's physical strength is important to both the manual and automated tool.
AI is a multiplayer to the user's intelligence, as the jackhammer is a multiplayer to physical strength.
>I'm not sure I want to live in a world where thinking is a gym like activity
That kind of describes the experience of retired people who do sudokus to stave off dementia. I suspect it's a bit akin to going from being a lumberjack to doing 10 squats a day though.
Engine scales without limits. There is no mechanism by which more engines in use lead to worse engines (leading to less use of engines). LLM output quality suffers heavily as the percentage of human-generated input drops. Which is what is happening right now as humans hand content production to machines.
> Just as the engine replaced physical strength, artificial intelligence, through models like large language models, is now replacing cognitive labor and thought.
Curious how the "AI" going to replace thought, as it is incapable if creating anything. Its just statistically matching data patterns for G-d's sake, don't make it something anything beyond that
meanwhile you're not even doing it statistically, you're just (deterministically) parroting the same argument that was already bad taste 10 years ago
quantum mechanics is just linear algebra, chemistry is just quantum mechanics, biology is just chemistry, and thus human intelligence is just linear algebra, so what's the problem? what else do you want from a mathematical framework, other than the ability to describe the system you need to describe?
Nvidia literally markets H100 as a "GPU" (https://www.nvidia.com/en-us/data-center/h100/) even though it wasn't built for graphics and I doubt there's a single person or company using one to render any kind of graphics. GPU is just a recognizable term for the product category, and will keep being used.
I assume someone is doing rendering on them given the OpenGL support. In theory, you could do rendering in CUDA, although it would be missing access to some of the hardware that those who work with graphics APIs claim is needed for performance purposes.
It's a good question. I'll note that, even in the GPGPU days (eg BrookGPU), they were architecturally designed for graphics applications (eg shaders). The graphics hardware was being re-purposed to do something else. It was quite a stretch to do the other things compared to massively-parallel, general-purpose designs. They started adding more functionality to them, like physics. Now, tensors.
While they've come a long way, I'd imagine they're still highly specialized compared to general-purpose hardware and maybe still graphics-oriented in many ways. One could test this by comparing them to SGI-style NUMA machines, Tilera's tile-based systems, or Adapteva's 1024-core design. Maybe Ambric given it aimed for generality but Am2045's were DSP-style. They might still be GPU's if they still looked more like GPU's side by side with such architectures.
GPUs have been processing “tensors” for decades. What they added that is new is explicit “tensor” instructions.
A tensor operation is a generalization of a matrix operation to include higher order dimensions. Tensors as used in transformers do not use any of those higher order dimensions. They are just simple matrix operations (either GEMV or GEMM, although GEMV can be done by GEMM). Similarly, vectors are matrices, which are tensors. We can take this a step further by saying scalars are vectors, which are matrices, which are tensors. A scalar is just a length 1 vector, which is a 1x1 matrix, which is a tensor with all dimensions set to 1.
As for the “tensor” instructions, they compute tiles for GEMM if I recall my read of them correctly. They are just doing matrix multiplications, which GPUs have done for decades. The main differences are that you do not need need to write code to process the GEMM tile anymore as doing that is a higher level operation and this applies only to certain types introduced for AI while the hardware designers expect code using FP32 or FP64 to process the GEMM tile the old way.
How long until a "PC" isn't CPU + GPU but just a GPU? I know CPUs are good for some things that GPUs aren't and vice versa but... it really kind of makes you wonder.
Press the power button, boot the GPU?
Surely a terrible idea, and I know system-on-a-chip makes this more confusing/complicated (like Apple Silicon, etc.)
Never. You can to a first approximation model a GPU as a whole bunch of slow CPUs harnessed together and ordered to run the same code at the same time, on different data. When you can feed all the slow CPUs different data and do real work, you get the big wins because the CPU count times the compute rate will thrash what CPUs can put up for that same number, due to sheer core count. However, if you are in an environment where you can only have one of those CPUs running at once, or even a small handful, you're transported back to the late 1990s in performance. And you can't speed them up without trashing their GPU performance because the optimizations you'd need are at direct odds with each other.
CPUs are not fast or slow. GPUs are not fast or slow. They are fast and slow for certain workloads. Contra popular belief, CPUs are actually really good at what they do, and the workloads they are fast at are more common than the workloads that GPUs are fast at. There's a lot to be said for being able to bring a lot of power to bear on a single point, and being able to switch that single point reasonably quickly (but not instantaneously). There's also a lot to be said for having a very broad capacity to run the same code on lots of things at once, but it definitely imposes a significant restriction on the shape of the problem that works for.
I'd say that broadly speaking, CPUs can make better GPUs than GPUs can make CPUs. But fortunately, we don't need to choose.
With LLM generated code (and any code really) the interface between components becomes much more important. It needs to be clearly defined so that it can be tested and avoid implicit features that could go away if it were re-generated.
Only when you know for sure the problem can't be coming through from that component can you stop thinking about it and reduce the cognitive load.
Regarding some of the ‘layered architecture’ discussion from the OP, I’d argue that having many modules that are clearly defined is not as large a detriment to cognitive load when an LLM is interpreting it. This is dependent on two factors, each module being clearly defined enough that you can be confident the problem lies within the interactions between modules/components and not within them AND sharing proper/sufficient context with an LLM so that it is focused on the interactions between components so that it doesn’t try to force fit a solution into one of them or miss the problem space entirely.
The latter is a constant nagging issue but the former is completely doable (types and unit testing helps) but flies in the face of the mo’ files, mo’ problems issue that creates higher cognitive loads for humans.
You just described 90% of coding