I'm probably a bit late on this, but none of these points touched on a software engineer building a complex architecture for no reason. I have seen this so many, many times. Yes, they can handle point #1. They take all the tickets associated with it and solve them quickly because they built it and know all the connections. The implementation makes no attempt (or claim) at scalability so that's a moot point. For point #3, interestingly enough these are never about optimization of anything, but about abstraction over abstraction over abstraction. As for the graph DBs, those aren't allowed anyway by the company they work for (thank goodness) or they would find some way to incorporate them.
Yes, I'm being a bit glib, but over-engineering does not always mean there's some good intentions in there. It can just be really bad.
graph dbs are just easily traversable joins of all many-to-many tables. although they have way less tooling in most ecosystems to support them (e.g. REST API was tough to do), they are extremely powerful [realizing i should write a post]. so, i would only use the entities i was modeling were natural graphs: chemistry, supply chain, network.
Build it the stupid way first is short sighted imo but this might just be an exaggerated way of saying premature optimisation is wasteful.
Just putting it out there, but I think Knuth’s aphorism is being used and abused to the point that some folk believe system qualities are second to functional requirements when it’s not the case.
I've been the victim of this, a producer consumer system with a single index file using some variant of a heap.
For each incoming request the producer;
locked the file, read the heap from the file, added the value to the heap, then wrote it back and unlocked it.
For each consumer thread, the thread did the reverse. It was setup with threads created by incoming requests, and a threadpool of 128 threads for the consumer.
This worked seamlessly until one day, at 3am PST, a flood of requests came in and the index file grew to ~4mb, and everything ground to a halt.
The legend of the architecture was a senior engineer did this on a weekend, then a month later left the company.
It isn't about making an architectural design decision / change at 3am, but someone has to live with the consequences of such a design decision and fix something in some way or another at 3am in the morning.
I.e. you could go with a clustered db setup which is supposedly HA, or a classical single instance DB...
The point here is, whoever has to get up at 3am to fix a potential catastrophic failure in the setup should have a say in it.
If the architect is also the coder/support, then it's a moot point. But if there is a separate architect, or a pack of them, there's no point in waking them up at 3am. Any architectural change isn't going to be deployed within the hour.
Personally, I think the designers/architects should work on the code too. You have to dogfood your own design, so to speak.
