I have a hard time knowing if I am going to need something in the future or if something is "YAGNI."
I'm not sure if the issue is more common in web applications than in other types of applications, but every choice I make feels like a catch-22. If something is not modular enough, then I may regret one day. If something is too complex, I may also regret it one day.
One way I've seen architectures get really complex really quickly, is when engineers try to foresee and plan to multiple rare non-critical scenarios.
The famous "what if" driven architecture:
- What if we get 10k concurrent users tomorrow? Everything needs to be CQRS event-driven, HA queue backed, produced and consumed by a fleet of microservices or it wont work perfectly!
- What if the user loses their internet connectivity while submitting this mundane form? We need localstorage backup for all CRUD operations and implement custom conflict resolution logic for each POST/PUT/DELETE.
I mean, yes we can try to tackle all possible issues and outcomes in our timeline but the sun would have cooled down by the time we are done.
Sometimes, well most of the times, you just gotta keep the project focused and 80/20 it.
Last year I did an interview and during the "systems design" phase I was asked to design a video streaming platform for 500 million simultaneous users. For a platform that has maybe a few thousand users, and is very unlikely to every get more than 10,000s users because it's just not general enough.
I never did find out if the test was to recognize that the number was off-the-walls crazy and that this would be the largest platform on earth by a substantial margin, or if they really thought that was a reasonable number. I suspect it's the latter :-/ I didn't get hired anyway, so...
I've also seen microservices for B2B software with low-hundreds of users. If the number of users is within an order of a magnitude of the number of services, then your developers are in need of adult supervision and a spanking.
Got into a similar argument the other day... The lead Architect wanted to create temporary ids so that a foreign system could pull a request only once, after they'd been notified so as to prevent "overload" of requesting every account every hour as a counter example... "so like 40k requests an hour?" BFD.
I had at least won that one... the id's will be date driven, natural keys per account and return a 404 if it doesn't exist yet. They'll still get a notification (web-hook) when the event for creation happens, but still far less complex then temporary id's that only work once, and require manual intervention if something breaks and needs to repeat.
I absolutely hate unnecessary complexity and tend to fight it at nearly every step.
If I didn't need the income, I'd be working on an email/calendar platform to replace Outlook+Exchange/O365. It's a space begging for disruption and not enough focus on some of the more critical needs that they provide that competitors tend to miss. Google comes the closest IMO.
100% I believe this is the most common pitfall that I've seen. Long-term thinking is an acquired skill when it comes to engineering good solutions. Unfortunately after (sometimes) years of over-engineering solutions in your career, the "a-ha" is that you definitely wasted your time and worse: often made iteration based on actual requirements slower by creating flexibility in one preconceived way, whilst adding rigidity in many others. KISS gets you real far
>"The Architecture Astronauts will say things like: “Can you imagine a program like Napster where you can download anything, not just songs?”"
Yes, and that is how things like VPNs, Tor, BitTorrent (but more broadly, P2P file sharing/downloading), Bitcoin (but more broadly, cryptocurrency/blockchain) and (more recently) Decentralized (aka "anti Big Tech censorship") Video/Content Sharing -- were all invented...
I think the counter to this would be that if you focused on the "listen to a song I want right away" problem - you would have invented Spotify. From a business perspective inventing Spotify is probably better than inventing bittorrent.
Elegant solutions are lovely, but often miss the problem people wanted to solve. Sometimes a useful product is built on top of an elegant solution e.g. WireGuard vs Tailscale.
Seems similar to academic research vs engineering. Both useful, different goals/outcomes.
I find the peer-to-peer example odd, because in the context of music sharing, p2p had a very simple, nontechnical purpose: Build something that no business would be interested in building, because they'd immediately be targeted by the MPAA and RIAA. P2P had the pretty clear-cut purposes to allow Napster to reach a massive scale without the technical infrastructure that would have been required for that (and could only have been provided by a company) and to make it immune from being taken down.
So the secondary purpose of P2P was bootstrapping, the primary was to evade the law.
Later, after the music industry learned their lesson, a business like Spotify could emerge that cooperated with content producers. Of course then they didn't need any exotic architecture anymore, because the primary reason for that architecture was gone.
The cool part of Napster (and the value added by the p2p architecture) was that it let you decouple the problem of delivering songs on the computer from the much more difficult problem of having to license them.
All they’ll talk about is peer-to-peer this, that, and the other thing. Suddenly you have peer-to-peer conferences, peer-to-peer venture capital funds, and even peer-to-peer backlash with the imbecile business journalists dripping with glee as they copy each other’s stories: “Peer To Peer: Dead!”
for the simple fact that it’s a reminder that if we have indeed strayed from the light, we did so long ago.
I find myself contrasting "peer-to-peer" to other fads/phases that had varying success (microtransactions, social networks, cryptocurrency, gig-economy, LLMs) and I think I still have a soft spot for it, probably because it represented a kind of individual empowerment.
This was also back in the day when the average computing device felt like capital accessible to the common man, machines which could be wrangled into solving your own problems on your own terms.
That's in contrast to a curated service graciously permitted by The Powers That Be, which doesn't do anything that's good for you that isn't also good for them.
Blockchain madness, crypto madness, AI madness, I wonder what's next.. I got used to hype cycles by now and am very skeptical of what they say. It's more interesting to see what ended up staying with us after the hype cycle subsided...
I feel like I may have missed the whole Single Page Application (SPA) boat. I am not saying that SPAs are still not popular or anything. But rather, I have started to see more push back against them or more people starting to look into alternatives like HTMX, for example.
So, I never took the time to learn SPAs and my employer doesn't use them. At times in the past, I would be lying if I said I didn't have a bit of FOMO. However, if the pendulum truly does swing away from SPAs, then I might feel a bit vindicated in my apathy.
