One term that captures a sizeable slice of the discussion is Emergence[1]
There is some size tipping point (waves hands) where the Toy Problem exits the nest and starts encountering the Real World. The really good RDBMS design encounters data at scale that exposes fresh pain points, and so forth.
The better engineers will have a fighting chance of surviving the onslaught.
Gall's Law:
"A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system."[2]
Very few clients want to pay for certified defect free software, as they prefer the lesser cost (and greater speed) of defective software that’s patched as defects become problematic. So most software exists without major defects and in a way that repairs are minimally expensive.
Software engineering isn't about perfection. It's about software that adequately meets the need, which is a lower standard. And cost and time are part of that "adequately meets the need". Perfection would be wonderful, but if it takes forever and costs infinite money, then we're better off with something we can use and afford now. (The rigorous pursuit of perfection, on large systems, means no software, ever. That's not a net win.)
So there's a local maximum in the value curve, between "worthless because it's too buggy" and "worthless because it's not finished". Hitting that sweet spot - or close - is what software engineering is about - delivering software that's actually usable and useful in the real world.
Yeah, but for many things, I don't need it. Do I need a secure word processors? (I mean, yes, in some environments I need a word processor that can operate on classified documents and be guaranteed to not make any copies anywhere else, or make any network connections, or leak in any other way. For most of my life, though, I just need a regular word processor.)
Which brings me back to my point. If I have to get a fully, 100% guaranteed secure (and also crash-proof) word processor, I'm going to be waiting a long time, and it's going to be very expensive. Meanwhile, I've got stuff to write today, and my budget isn't any too fat as it is.
Sometimes fully secure (or man-rated, or 9 nines, or fully validated, or whatever) software is what is needed. Much of the time, though, it's a better engineering trade-off to not go that far.
