Have a look at a newsletter called "Big Tech Digest". It provides you with the latest articles from hundreds of engineering blogs like Meta, Google, Uber, Airbnb, Doordash, etc. every two weeks. Very useful for building knowledge around system architecture.
Also, often choosing the 'right' technology is less important than designing an architecture that is both simple enough and capable enough.
When doing this in my career, I am always considering can it be more simply (i.e. less complexity is easier to grok and is usually less prone to fail) and can it be done more clearly (i.e. such that the intended use of the architecture is obvious).
I don't work in the field so I can't say for certain, but AWS and friends really strike me as the type of tech you need to deploy at scale in order for their complexity to be worth the initial up-front learning cost.
Try to think of a project that either needs infra like that to get started, or another one you can deploy in a homelab or a few VPSes or something.
Maybe VMs are a good idea, to give yourself a good place to test things without committing bare metal to anything.
What software system architecture interests you? The scale you're interested in may matter, too. For me, system architecture means which distro (and which set of packages), configs, and other settings that create a cohesive server/client or groupware system. I think a lot of those challenges lay mostly in understanding how computers are being used within a given organization, and designing things in ways that make adapting easier and workflows smoother.
What that looks like, I would imagine is different for every organization.
Some of it is opinion and personal preference, some of it is respecting and adhering to patterns and techniques already in place. All of it requires some understanding of the pros and cons of different technologies and architectural decisions.
I'd be surprised if you could simply read a book or watch a video and learn how to architect complex software solutions, honestly. For me working at different orgs and being curious about how all the various layers of the software stack fits together has given me a decent perspective on different approaches to problems.
Also, "good" here is somewhat relative. Being "good" an architecting problems for large orgs operating at scale is far more challenging than being good at architecting solutions at a small startup.
Smaller, younger companies generally don't need anything super fancy or complex, what's important is getting things done so typically you build simple and worry about doing things right later. Some basic AWS knowledge and a strong coding background is probably all you need.
Larger, older orgs are far more intimidating because there's so much complexity already in place, and often there will be entire teams built around managing that complexity. In these cases you need have an appreciation for what already exists and build things that complement and improve on that. You also need more experience with building at scale. At a small startup whether you decide to go serverless or not doesn't really matter, but at a large org with tens of thousands of daily users the cost of a bad decision grows exponentially.
Another thing I'll note is that if you're doing this for career opportunities whether you can do something doesn't matter one bit. Companies won't hire you unless you have a track record. If you want to become a technical architect I think the challenge is mostly knowing how to play the game in such a way that you can take on those responsibilities at the place you work. Learning how to architect new solutions isn't that difficult if you're curious about how things work.
i would say experimentation. trying implementing different arichtectures with relatively simple implementations. one thing that helped me was getting books and articles about different design principles architectural ideas etc. and then creating them myself. Also, what is 'good' here? i am assuming, knowing when to use what kind of architecture. some are easier to create, some perform better, some are easier to collaborate on in large teams, costs can be a factor too. Are you looking for a specific thing or wanting to generally know more about the topic.?
I have a specific thing in mind, but I think I lack sufficient knowledge of what's out there to make the best trade off for my specific use case. To be honest, I lack basic knowledge of different system components. To give an example I know redis is a key value store. But i wouldn't know the drawbacks of using it and what usecases its best use is for. I would like to broaden my knowledge in order to answer such questions as my job requires me to go from a data scientist to more of a solutions architect
"Gall’s Law: all complex systems that work evolved from simpler systems that worked."
Do the simplest thing that works, if that is hosting something on a VM with SQLite for persistent storage than that is what you implement.
If what you had in memory was a map or a queue and you don't care about persistence then redis is likely going to serve you well when you need to scale to more than one instance, if you never need more than one instance then keep it in memory as a map or a queue.
Don't add things "because that is what :bigtech: does" :bigtech: is not you and if it was, you would have the team to do it or would be building the team and not the software.
There's a balance here. To paraphrase the parent: Don't spend much effort solving problems that you don't have yet, and may never have.
On the other hand, try not to paint yourself into a corner, where you can't grow (if you do) without a major restructuring of the architecture. This takes having some idea of what the problems can be, and what people do to handle them. You don't have to implement those things yet, but try to build what you build in a way that will be friendly to those changes if they happen.
I don't disagree but I generally do this by putting thw abstractions in place ij the software.
An example is for crud, use the (onion|hexagonal|whatever you want to call it) architecture, you business logic has nothing to do with how the data is stored persistently (or not), build your application using abstract APIs.
It's slightly more code up from but if I was using a map in memory it is trivial to move to using redis if I build with the abstractions in place to begin with.
Generally in the prototype stage I won't care about that either but I expect the prototype / poc code to be deleted.
Same idea with SQLite, moving feom sqlite to a full blown database isn't that difficult but this is one of the places where I would conceded and say if you have the infra in place just use what is available, it may be easier to get a DB on thr current cluster in the long run if there is already a cluster that you can get a DB on.
If you however are in a completely greenfields position and really have no idea where the product is going, sqlite scales to a decent amount of RPS and so does a simple go/rust server on a host.
Many webservices really don't need more than that and not worrying about being distributed when you don't know if you ever do need to be distributed makes whole hosts of problems trivial to solve and allows you to use technologies that would usually be inconvenient.
Learning by doing is obviously the best way, but not every role has that kind of opportunity. You can build something on the side and scale it, but it's easier to just do that in a company :)
I'm no expert on system architecture, but I have noticed that reading a lot of docs and blogs on how other companies do things successfully has helped me:
Quastor: quastor.org - Engineering blog posts and summaries
Engineer's Codex: read.engineerscodex.com - Case studies and summaries of Big Tech systems and software architectures
SWE Quiz: swequiz.com - Bunch of useful questions that has helped me a lot at work
TLDR Web Dev: tldr.tech/webdev - Good for keeping up on engineering-specific stuff in general.
ps. I'm the author :)