
Ask HN: How to get good at designing big and complex applications? - rochak
Hi HN! I am a grad student with a focus on CS, specifically Software Engineering. I like to explore different technologies that are used to build a product and am fascinated by the way people design such big complex systems. For instance, I am taking a class on Operating Systems right now which also requires me to develop my own kernel. I am finding it very exciting, challenging and fun. I would really like to know what all I can do to get better at designing such big complex systems. I try to read open source but it always feels like a challenge. I don&#x27;t know if I have the aptitude for this but studying CS and the thought processes of people who have spent so much effort in this field gives me a lot of joy. So, if you have any advice, resources, books or tutorials that you think might help me, please do share. I would be really obliged. Thanks!
======
bg4
"A complex system that works is invariably found to have evolved from a simple
system that worked. The inverse proposition also appears to be true: A complex
system designed from scratch never works and cannot be made to work. You have
to start over, beginning with a working simple system."

~~~
gavinray
I just made the connection between your statement _" A complex system that
works is invariably found to have evolved from a simple system that worked."_
and the common understanding among experienced developers that proficiency in
software and the primary goal when designing and building a system is
perpetually fighting complexity.

You design and write the cleanest, most simplistic & adequate solutions for
problems as they arise and ruthlessly battle introducing complexity as
features and scope grow.

Inevitably, after many years you will wind up with a system which by sheer
size is complex. But only reason it works is because of the never-ending
battle in the fight against complexity.

~~~
bg4
Yes, have to relentlessly drive down complexity. Starting with simple, pure
functions that work and composing them together to build up more complex
functionality goes a long way.

------
rasikjain
This comes with experience. There is no single answer to get good at the
designing complex systems.

The more you work in different applications, the more you discuss with your
peers in your organization and other meetup groups, explore different design
patterns (Martin Fowler), Follow well known Architects or Company engineering
blog [0] posts, Try doing proof of concepts etc. Eventually as you progress
with experience, I am sure you will get that knowledge.

I used to follow the below link [1] to understand how the systems work at
different companies globally (e.g Facebook, Google, Netflix, PlentyofFish
etc). This has very interesting content.

[0] [https://github.com/kilimchoi/engineering-
blogs](https://github.com/kilimchoi/engineering-blogs)

[1] [http://highscalability.com/all-time-
favorites/](http://highscalability.com/all-time-favorites/)

~~~
rochak
Thanks for the advice and the links! Also, I came across a book called
Designing Data Intensive Applications. It was very highly rated on Amazon and
on HN as well. Do you think it would help me? It explained a lot of concepts
but after reading for a while, I found myself not following it. So, I decided
to tackle each topic from scratch.

------
aynyc
Do people find Martin Fowler and his peers' work to be good in real world? I'm
a system/application architect who spent years coding. I used to read books by
Martin Fowler and others religiously. But now, I realize most of the books and
talks are really just fluff pieces because the examples worked in perfect
condition, where as my experience is from the battle field.

As far as getting good at designing big and complex application, I'll give
what I know.

1\. Everything complex came from a small system. So make sure your small
application works first. Twitter/Google/Facebook didn't get to what they have
now on day 1. They failed a lot to get here.

2\. Designing monolithic system is a lot easier than micro services.

3\. Read a lot of codes and how different component interact with each other.
For Python, projects such as Scrapy, Flask and Django are good example to
start.

~~~
rochak
Thanks! Iteratively improving and adding features seems the most natural way
how a software matures or ‘becomes complex’. I guess I just have to start
building something small and learn how to add features that require learning a
new concept.

~~~
aynyc
Your first step should be avoid complexity. Reject features (big or small)
that don't add value to the system. And if you must add feature, try to make
it as specific as possible. Generic features tend to get abused. I personally
don't like it, but other do love it.

~~~
godshatter
I would add to this that you shouldn't be afraid to refactor when you hit a
wall where you can't add the feature you want to add because of the way the
other features structured your code. Better to take the time to modify the
basic structure a bit to make it work than to try to kludge something together
to sort of make it work.

You'll find that doing it right allows other features to be added that you
haven't thought of yet. If you somehow force it to sort of work, you'll end up
doing that for a lot of other features, too.

~~~
rochak
Now that I look back, that is how I have approached developing systems and it
certainly makes a lot of sense.

------
kaazhan
Hi,

There's no single way. In fact, beeing a good software architect is all about
having many ways in solving a problem, knowing wich solution is the most
adapted depending on the context. All is about culture and cleverness To
improve, you can :

\- Follow/read the well know architects. For example, I read this blog
[https://martinfowler.com/](https://martinfowler.com/) and watch most of the
conferencies he gives.

\- Learn the design patterns and the architectural ones.

\- Read a lot of code, like github repos. It does not care if you're reading
repos on a subject close to the one you're interested to. You need to see a
lot of architectures to see some clever solutions some peoples have used to
solve some specific problems. I don't think there's any shortcut.

\- get interested in meta development. Like "why does the languages work the
way they work ?", read about how code is produced in real conditions (what are
the processes involved), etc.

~~~
quickthrower2
I’d also add go to excellent meetups with high quality talks. They attract
good people, get talking and you’ll have 100 things to google when you get
home!

~~~
rochak
What to do when those talks are too complex for my understanding?

~~~
quickthrower2
They are for me too. Ask questions after the talk. Get the gist if you can
even if you don’t grok all the details. Speakers often go too fast through
stuff so I don’t take it personal if I don’t understand it all or any of it.

~~~
rochak
Thanks! For some reason I find it difficult to just approach people and talk
about stuff with them.

~~~
kaazhan
You're still a studient. It is not possible for you to have a good knowledge
of software architecture because it is a really complex domain. If you show
interest in it, read about it, go to conferences and ask questions, don't
worry. Peoples will act cool with you. Anyway, the only way to progress in
dev/software archi/devops/etc is to put your ego aside and accept the fact a
lot of peoples knows whay more than you. If you go speak with them and they
treat you like garbage, this is their fault, not yours.

If you're really shy/impressed, maybe you can go with some friends and speak
about it together. It will help you feel like you understood some things.
Architecture is not filled only with youg peoples, and if you're a group of
20+ y/o speaking together, eventually the older ones will come and speak to
you. then you can ask them what to read/watch to have deeper understanding

------
matijash
Congrats on your will and enthusiasm, I believe this is already a great first
step!

Besides Martin Fowler which was already recommended, I would also recommend
the books by Robert C. Martin (Uncle Bob), and his "Clean" series - there are
books "Clean Code", "Clean Architecture" \- all of them are really good and
useful and will give you a solid foundation to build on.

Other than that, the best thing you can do is to get practice & ideally
surround yourself with more experienced developers. Either get an internship
in a company building something more complex, or participate in e.g. Google
Summer of Code.

Doing fun side projects is also a great way - you learn a lot and also have
something to show & write about - win-win!

~~~
gentleman11
I have been looking at clean architecture and trying to apply it to a Unity
game. My impression is that industry wisdom favours a lot less OOP and a lot
less abstraction in their architectures (eg, Jonathan blows talks, hundreds of
posts on Reddit), mostly for performance and velocity reasons.

Do these techniques apply to games, or are they the wrong tool for the jobs?

~~~
rochak
Have you checked [0]?

[0]
[https://gameprogrammingpatterns.com/](https://gameprogrammingpatterns.com/)

------
avinassh
Check the books of Architecture of Open Source Applications -
[http://aosabook.org/en/index.html](http://aosabook.org/en/index.html)

~~~
rochak
This is a good find! Thanks!

------
tilolebo
This book (Designing Data Intensive Applications) was an interesting read:
[https://dataintensive.net/](https://dataintensive.net/)

~~~
rochak
Thanks! I knew about this book before but couldn’t go through it a lot because
it started to get very tough. So, I decided to read each and every concept
from the basics before I jumped into that book again.

~~~
tilolebo
It is tough indeed, especially if you don't have a use case for it.

Experience > theory for such a topic. Go ahead if you find it interesting, but
don't expect to become an expert on the topic just by reading about it ;)

~~~
rochak
That is also true!

------
segmondy
Study & Experience, that's it.

Also understand the difference between these

    
    
         programmer
         application developer
         software engineer
         computer scientists
    

there are very different roles, it's okay to be one and really good at one.

BTW, you mentioned "design" so I'm going to hinge on that, design not the
actual implementation. In that case study existing systems. 90% reading can
teach you design, on actual implementation, more like 90% doing.

~~~
rochak
What if I’m not inclined to just one of these? I like learning things and
figuring out why which is why I see myself fitting into every category.

~~~
segmondy
Nothing wrong with that, if you are a generalist you will find yourself
intersecting those. Else you will find yourself focused in one. There are very
few organizations in the world where you get to do all of those, except for a
startup in the initial stage or by you pursing your own side projects.
Something like game programming might intersect all (programming - using GPU,
app development - the game engine, software engineering - building a scalable
game server, computer science - utilizing appropriate algorithms and data
structures that most people will tell you they never use in the real world)

if you want to do the programming thing - learn low level programming, buy an
arduino etc and learn to program it.

app development - build websites, mobile apps, desktop apps.

software engineering - contribute to a large scale app (kubernetes, postgres,
etc)

computer scientist - study latest CS papers, replicate result, write your own
paper and come up with your own ideas.

~~~
rochak
After you define what these terms mean, I think I am more inclined towards
programming and software engineering. Thanks!

------
subhobroto
The early decision you will have to make is whether you want to be a
theoretical specialist who will mostly make money consulting working at a
consulting firm or university or whether you will be solving actual business
problems down to the metal.

Each has a completely different trajectory at this point in your life.

A great example is to read the great debate between Andrew Tanenbaum and Linus
Torvalds over kernel design as its closely related to what you're currently
studying.

Linus would have clearly failed Tanenbaum's OS class.

Yet I use Linux every day, as does millions of others, while Tanenbaum's MINIX
is on a CDROM that came with the book that I bought for my OS class.

This is not to say Linux is still a monolith, but that would be getting in way
too much details.

~~~
theandrewbailey
> Yet I use Linux every day, as does millions of others, while Tanenbaum's
> MINIX is on a CDROM that came with the book that I bought for my OS class.

You sure about that? Do you use an Intel CPU?

[https://www.zdnet.com/article/minix-intels-hidden-in-chip-
op...](https://www.zdnet.com/article/minix-intels-hidden-in-chip-operating-
system/)

~~~
subhobroto
Andrew, I didnt know that. Thank you.

Now, let me ask you this.

If someone was to ask you either of:

1\. "Between MINIX and Linux, which is a great example to study as an OS
that's deployed on billions of machines worldwide and used by millions of
engineers?"

2\. "Between MINIX and Linux, learning administering which, offers me the best
job prospects in the general tech industry?"

What would be your answer in each case?

------
ooooak
> How to get good at designing big and complex applications?

Write small and simple programs.

