
Ask HN: Is it worth getting into distributed systems after 12 years full stack? - sidcool
I have done 12 years Full Stack engineering, including some DevOps.  Recently I have been getting very interested in Distributed Systems, and not just at a consuming level but at the theory level.  I would like to get into the gritty details of distributed systems, perhaps to the level of a Masters&#x27; Degree.<p>Is it worth getting into the the subject?  It would take a major commitment in time and efforts.  Would it help with better career opportunities?
======
ArtWomb
Yes, most certainly. Best way is to look at cloud technologies used today:
AwS, Google Cloud, Azure. Cloudflare's tech blog is also very good.

One growth area is cloud gaming. And I think "virtual desktops" in general may
finally gain some consumer traction and scale pretty soon.

Naturally this leads to an interest in Decentralized Applications as it
represents an opportunity to re-build the Web.

Some resources to get started. Best of luck ;)

MIT 6.S974, Decentralized Applications, Fall 2018

[http://nil.lcs.mit.edu/6.S974/papers.html](http://nil.lcs.mit.edu/6.S974/papers.html)

Web 3.0 Summit Playlist

[https://www.youtube.com/playlist?list=PLxVihxZC42nHlc7J-UqeS...](https://www.youtube.com/playlist?list=PLxVihxZC42nHlc7J-UqeS-
asagbehBeh3)

~~~
ozfive
Virtual desktops will open up opportunities of virtual phone operating
systems. If you leave your phone at home by accident pick up a "shell" at the
grocery store and sign into your account. When your done with your errands
drop it off at the last place you were at, or some other concept. Truly
portable number.

~~~
Nydhal
Until no one actually owns phones and all you have are shared shells that can
be picked up anywhere.

~~~
theoh
Unfortunately, as with the idea of batteries for electric cars that can be
swapped out at the charging station, there is an incentive to hold onto better
units, and quickly return inferior units to the pool. It's a case of Gresham's
Law.

So the utopia of interchangeable shared units of anything will only be viable
if those units are completely indistinguishable. If there's any kind of
distribution of quality, the incentives will cause the economy of the system
to devolve into hoarding of the better units. One way to work around this is
to mandate regular exchange, but that really gets you into the territory of
communism. Not necessarily a bad thing in theory or in some vastly different
future, but a big change.

~~~
ozfive
You would have to swap out stock anyway as all physical things degrade with
use. It would be a natural culling of devices that would need to be pulled
from circulation regardless.This idea doesn't exclude owning your own more
powerful device that may have been left at home or a friend's house. You're
not interested in the physical object and keeping it for long periods of time.
It's a short-term convienince proposition.

------
defined
I have been working in distributed systems for 6 years or so. It is a field
ripe with difficult problems and active areas of research. Some things people
don’t even think of as distributed systems suffer from many of the problems
inherent in that field.

For example, a system using a database with asynchronous replicas - especially
if latency is not extremely low - using those replicas for read scaling, is an
eventually consistent system (because of replication lag) and suffers from
related issues.

Multi-master databases are even worse. You get the joys of network
partitioning (netsplit or split-brain) and conflicting updates.

Sure, there are ways to work around this, but all the workarounds I know of
have some rather ugly tradeoffs.

Basically, there are many opportunities to do something great, and not be
bored. IMHO.

~~~
sidcool
100 times this. The replicated DB with master-slave and the CAP theorem was
something that got me interested in Distributed systems in the first place.
Google Spanner, I feel, is a marvel of modern tech. Thanks!

Do you recommend a full time degree in DS or actual projects?

------
weliketocode
Are you specifically targeting better career opportunities?

Now is as good a time as ever to be a seasoned Full Stack engineer. Pursuing
an MS and shifting specialties seems very expensive.

Despite all the support other commenters are giving, I believe you should be
very hesitant about making such a career shift.

~~~
WalterSear
Fwiw, my experience has been that many of the job openings spurring the growth
in advertising for Full Stack engineers are actually glorified Front End
roles, with very little work at the interface or on the back end.

I suspect that, due to bias against Front End developers (and javascript :) ),
hiring managers are have trouble justifying senior front end developer
positions and so are resorting to overqualifying the position and unicorn-
seeking.

There's also a bias in technical interview questions, since conventional
algorithmic/puzzle questions hold less relevance on the front end, so
qualifying experienced engineers without asking back-end appropriate questions
poses difficulties. Which means that, despite a role being a Front End
position, Full Stack candidates end up looking like a better fit, since they
do better in the interviews.

~~~
gaius
_growth in advertising for Full Stack engineers are actually glorified Front
End roles_

Right. People think I’m a hater because I say “full stack engineer” is just
title-inflation for webdev, but I’m just telling it how the entire rest of the
world sees it.

Throughout most of the history of the web, the same people wrote the CGI
scripts, and any SQL in them, as wrote the HTML; working this way is perfectly
normal for a webdev for like the last 20 years. I guess we were all “full
stack engineers” back then, just we didn’t have a grandiose title.

~~~
WalterSear
IMHO, it's got more to do with the expansion and change of the Front End
developer role, than direct title inflation per se. Now, a front end developer
has to maintain consistent behaviour across a panoply of platforms - each
under constant change and growth, whereas the scope of actual 'web
development' has remained a relatively static. It's becoming less and less
realistic to expect one person to be able to effectively handle both roles any
more, without making compromises on UI behaviour and compatibility.

------
mabbo
I think the concept of "distributed systems" is very broad.

Do you mean service oriented architectures? That's the practical 'distributed'
systems many companies are using today. You don't need a degree- just go work
at Amazon for a couple years.

Do you mean decentralized systems and algorithms? Paxos, Raft, Bitcoin and
TOR? Then sure, you might consider a master's degree. Or just start reading
papers online and messing around with code in your spare time.

------
xiphias2
I vote yes for distributed systems, no for the Masters' Degree. If you already
know how to code, it's easy to get into other fields and train yourself,
thereby not losing money. You can also write open source code if you don't
think you could find the appropriate job instantly.

~~~
ccajas
How much is it autodidact friendly compared to Full Stack engineering? What
would a "hello world" or "first learning project" of distributed systems even
look like? I can only think in terms of learning via building a concrete
project eg. a to-do app built on whatever tools you want to learn.

~~~
sidcool
Yes, there's no simple HelloWorld for Distributed systems. One must understand
a great deal of theory before committing to a role.

~~~
ccajas
Learn theory at a job then?

------
cevaris
Don't you need to know some of Distributed Systems to consider yourself Full
Stack engineer?

~~~
tedmiston
I think it depends on the scale of apps you build for. One can deploy a simple
backend API or web app to a PaaS like Heroku or Zeit, or even a PaaS-like
environment managed by another team, without knowing much about distributed. I
would consider someone that can do that + build a modern front end as full
stack.

IMO the average CRUD web app doesn't have the scale to have distributed
systems problems. Also there are managed service versions of everything these
days even when one scales up.

~~~
repsilat
Yeah. And the scales have to get pretty crazy before "rails on postgres with
some in-memory caching" stops cutting it. Your average developer will never
see it.

------
honestlyidk
"Is it worth getting into the the subject? It would take a major commitment in
time and efforts. Would it help with better career opportunities?"

I guess its an opportunity cost. What else would you be doing besides that.
with 12 years of experience Id hope that you have some understanding about the
subject already and arent completely in the dark. I guess with 12 years I
would say it should take you no time to pick it up cause I would expect you to
already be well into APIs and Oauth.

I wouldnt hurt your career opportunities. But I guess you could do something
more established like get a data science degree or work with embedded systems
to work with robots. Its really about what you want. But doing this is better
than doing nothing.

Technically I think that a lot of people do it for a cool factor rather than
for a need. You can add uncessary complexity without benefit. I dont find
there to be too much deep complex ideas about it.

Basically you are performing separation of concerns across a network vs doing
it by creating different class files, this allows you to offset query load to
an entire set of servers which helps scaling. But your company also has to pay
of another set of servers.

------
opportune
Yes it’s extremely interesting but just be warned: the hottest distributed
systems applications are at either extremely competitive fintech firms or
massive household name software companies, because at the end of the day,
those are the only groups that really need and can afford the state of the
art. After you get a masters, if you want to continue working on the bleeding
edge, those places are your options. So yes those are great career
opportunities but they may not offer what you want culturally

I work on / with what I think is pretty cutting edge distributed software but
outside of academia most of the work is, at the end of the day, still
engineering.

------
m00dy
I did my masters in Distributed systems and I think it was definitely waste of
time. I would prefer data science or something similar instead. Aside from
academic positions, there is not much opportunity in the market.

~~~
PopeDotNinja
> Aside from academic positions, there is not much opportunity in the market.

May I ask what you mean? I didn't formally study distributed system, but it
seems like much of my daily struggle as an engineer is trying to deal with
folks who are trying to implement consistent solutions in an eventually
consistent world. For me, distributed computing lies beneath every non-trivial
programming challenge I'm working on. Even OPs front-end JS talking to the
back-end is distributed.

------
mk89
I would say yes. What's the worst that can happen, you learning something good
and interesting? It's always good to have people in this field that know how
to design distributed systems.

------
zzzcpan
It's worth it, but probably not for better career opportunities. As almost
anything else for your career would have a bigger impact and would take less
time and effort.

------
dmourati
I don't believe Full Stack is a real thing. I also believe you have probably
been doing distributed systems even if you think you haven't.

~~~
honestlyidk
Well thats because you are not a full stack engineer. If you were you would
understand. Its pretty simple. You have backend , you have front end , you
have database. Full stack enginner has experience and implements features
across all these technologies. A bit of dev ops environment knowledge is
probably also good because you have to worry about the connection between all
these environments in order for you to do your work.

Sure there is a quality trade off, being jack of all trades but this industry
is very elastic. You can have a great frontend engineer with only 2 years
experience and an ok one with 6 years experience. Time in field doesnt really
gauge skill well. Its more about the challenges one takes on and the variety
of problems they solve with their code.

~~~
zerr
> Full stack enginner has experience and implements features across all these
> technologies.

I guess most of the 90s and early 00s web devs were full stack then, including
setting up the servers ("dev ops" I guess).

~~~
ThalesX
Well you had a tightly knit package back then, I'd argue that increased
complexity in the front end was what lead to the 'full stack' developer.

As the previous poster mentioned, it's quite simple, if you can implement a
feature vertically (and I'm thinking at least a 3-tier system should be
considered), you could consider yourself to be a full stack developer. I've
met plenty of developers in my career who refused to work outside their tier.

The fact that sales people and recruiters are making full stacks up to be some
sort of mythical horses with a single horn on their head is a different
story...

------
crocal
I would recommend getting a MS degree in /systems/ engineering. From there,
you can study the distributed world but also you can access the equally
fascinating world of general systems engineering. To make it better, your
computer background will be a plus compared to fresh grads.

------
postit
Short answer: Yes, but it depends.

Yes: I believe everyone would benefit from leaving their comfort zone and
venture into more DS to expose themselves to some non-trivial work. Don't get
me wrong, there are plenty of challenges on writing a working CRUD system from
the bottom up but facing scalability, consistency and fault tolerance issues
is a different word, and most software developers don't give a damn about
performance. That's usually what happens at startups or small companies(I've
been there several times). A small and focused team builds a working product,
and suddenly they need to bring more experienced engineers to help then scale
it once the cooking recipes from the internet aren't serving them well or
because the leadership won't let the team engage these challenges. So my first
advice goes here if your company has such problems try to expose yourself to
them.

Practice and theory should walk side-by-side, you can read the entire library
on distributed systems but still be mediocre when facing real problems because
you've never encountered those problems in practice. Use the books and papers
to give you a direction to where you should point and do your homework.
Consistently paced challenges and exercises gives you volume(consistency) and
helps you build experience. Build your glossary of concepts, nail your
difficulties and once you understand the basics, go read and implement CRDTs
in Rust by yourself :) - I'd also suggest you to try following a book on your
own pace first before committing to any formal education. Try to find a mentor
within your friends who can give you more advice's and peer review your work.

Now the sad part: Unfortunately when shifting expertise even within the same
career, you give up on several years of your experience. You will look less
appealing to recruiters and companies because you just turned yourself into a
JR. Distributed systems engineer w/o previous experience, and those are pretty
hard to find out there. If you can live with that(both checking our ego and
financially) and be humble enough to accept your new position - I'd recommend
you to embrace it.

------
matt_oriordan
Being part of a team who is designing and running a globally distributed
system has been the best 5 years of my 20 year tech career. Things go wrong
sure, and it’s hard, but is crazy rewarding and intellectually challenging in
all the right ways. I never quite got that when I was a full stack web dev.

I wrote a post on what a distributed system engineer is, at least from our
perspective at Ably Realtime. See [https://blog.ably.io/what-is-a-distributed-
systems-engineer-...](https://blog.ably.io/what-is-a-distributed-systems-
engineer-f6c1d921acf8) if you’re interested in what it’s like.

Matt, tech co-founder Ably, www.ably.io

------
imhoguy
Here is starter kit for you: [https://github.com/theanalyst/awesome-
distributed-systems](https://github.com/theanalyst/awesome-distributed-
systems)

Read, understand, repeat.

~~~
imhoguy
Ah and regarding career: Do Masters' degree if you want guided learning and/or
target an academic career.

But if you are self learner and want industry job with hands-on distributed
systems, I would recommend exposing any back-end/dev-ops/networking experience
in your résumé. Also obtain AWS and GCP certificates, they will require you to
learn about both theoretical and practical aspects of distributed systems.

Seek companies working on large scale distributed software, you may start as
full-stack or back-end engineer and then get closer to interesting projects.

------
jey
Yes, go for it, but definitely make sure you actually build something cool
during your Masters degree if you want it to improve your career opportunities
afterwards. The credential by itself won't mean much, but if you list some
interesting project(s) that demonstrate your new skill it'll help with future
job searches.

> Java, Scala, Ruby, Python, Angular.

You should also learn C. (Isn't that a pretty important part of the "stack"?
;-))

~~~
sidcool
Thanks! I am learning Rust nowadays. It's promising to replace C/C++. I just
don't know it enough to list it yet.

~~~
cyphar
Most core system software is still written in C though, and I think it's not
reasonable to assume we are going to rewrite more than 40 years of computer
history in Rust in any reasonable amount of time.

------
sidcool
Thanks everyone for the comments. I am going through each one and will reply.
This means a lot.

------
ilovecaching
Do what interests you and stop worrying about the consequences.

------
mettamage
> Would it help with better career opportunities?

It depends on the demand.

How many senior web devs are there? How many senior web dev positions are
there?

Guestimate: 278057 senior developers and 23050 open positions [1].

How many distributed systems people are there? How many DS positions are
there?

Guestimate: 1355 distributed systems people [1] and 219 open positions [2].

In other words: try to find some data that give some intuition. Sure, the
'research' (if you want to call it that, lol) is really biased but it probably
is better data than your own intuition if you have no clue about it.

\------- addressing some other stuff you said -------

Why do a masters in it? Do you want to research it as well? Writing a thesis
is the obligatory part of it. At the Vrije Universiteit Amsterdam there is a
masters in it that is spearheaded by Andrew Tanenbaum. But when you look at
the courses, you see that many courses aren't about the theoretical part of
distributed systems (see [3]).

To get your feet wet I'd recommend to fully follow distributed systems from
Maarten van Steen. Why? First of all, his book [5] is being recommended by
other people (see [6]). He also has lecture videos on it (from 2012, but since
it's from a theoretical perspective they are still relevant). Edit: he deleted
his videos, but I found one source here [4]. Oh yea, and he offers it all for
free (which is also why I'm comfortable posting the new URL of the DS videos).

These are my 2 cents. Goodluck!

My personal opinion, don't pursue a masters. Just scan study guides of
universities and figure out a curriculum, work less as a web deveoper but keep
doing it (i.e. 4 or even 3 days per week) and study it independently.

[1] I searched it via LinkedIn by typing "senior web developer" and "web
developer" and "distributed systems"

[2] I typed in "distributed systems" as job title on indeed.com. LinkedIn gave
nonsensical results since everyone's course Distributed Systems was counted
and almost no one had a title in it.

[3] [https://www.vu.nl/en/study-
guide/2018-2019/master/m-r/parall...](https://www.vu.nl/en/study-
guide/2018-2019/master/m-r/parallel-and-distributed-computer-
systems/index.aspx)

[4]
[https://u.cs.biu.ac.il/~ariel/download/ds590/videos/](https://u.cs.biu.ac.il/~ariel/download/ds590/videos/)

[5] [https://www.distributed-
systems.net/index.php/books/distribu...](https://www.distributed-
systems.net/index.php/books/distributed-systems-3rd-edition-2017/)

[6] [https://teachyourselfcs.com/](https://teachyourselfcs.com/)

------
bitxbitxbitcoin
I vote yes, but I'm probably a bit biased.

~~~
sidcool
What would your reasons be? I think we are all biased and I value strong
opinions.

~~~
bitxbitxbitcoin
Not everything in the world needs to be decentralized, despite what your local
snake oil ICO salesman might tell you. However, there are more than enough
significant institutions and processes that can and will function better with
a little bit of distribution, decentralization, federation, etc properly baked
in at key friction points. Building that future is a worthy and potentially
lucrative cause. I know I'm speaking incredibly broadly, but frankly I had a
similar question as you some years ago but without the benefits of being a
full stack engineer.

Disclosure: MSc in Digital Currency holder.

~~~
throwaway0255
I always thought distributed systems weren’t necessarily decentralized
systems. Is my understanding of the terminology wrong?

For example a system that uses distributed algorithms for performance
improvements, or a system that is distributed by design and necessity to
retain the benefits of redundancy, could be considered both centralized and
distributed.

~~~
bitxbitxbitcoin
Nope. Your understanding is spot on. Mine is the one that's flawed :P.

------
ilurkedhere
Which full stack?

~~~
sidcool
Java, Scala, Ruby, Python, Angular.

~~~
sixdimensional
Interesting. Some time ago I had the ongoing debate with a colleague on what
it meant to be full stack.

I was under the impression that full stack meant you can do what's needed to
stand up a solution at literally every OSI layer.

For example, from down at the base of infrastructure, networking,
hardware/cloud, servers / platforms, integration, data / database, custom and
COTs applications and services to clients, UI, desktop/web/thin/fat etc. etc.

By that measurement someone who can do all that probably already understands
and in a sense, works in distributed systems - you probably had to already to
cover the breadth and depth of all that knowledge.

Not picking on your choice of words here ("full stack"), just mulling it over
and thinking: 1) if you can do that you're qualified to work on distributed
systems and 2) there's no shortage of work advancing distributed systems and
that problem isn't going away any time soon.

I'd say distributed systems is challenging (I worked on a startup doing that
with database federation and it was really interesting and challenging), fun
and important.

I think maybe the bigger question is, are you interested and want to dedicate
the effort needed to get into it, and will that make you happy? There's so
many levels of working in distributed systems - from research, inventing /
building the tech, to implementing existing solutions (e.g. Apache Spark,
Kubernetes etc. etc.).

If that stuff is interesting I'd first decide if I wanted to be on more on the
research or practical side, and then how much money you need to live, and then
choose my entry point into the field.

~~~
wyldfire
> Interesting. Some time ago I had the ongoing debate with a colleague on what
> it meant to be full stack.

Yeah, that's what makes this term a poor one IMO. It's not at all clear what
it means. I've always taken it to mean "not merely Javascript."

~~~
NightlyDev
I think most people think of a fullstack developer as someone who is
experienced in everything from backend all the way up to frontend design.

Fullstack developer = someone who can do all parts of a project on their own
if they wanted to.

~~~
umanwizard
What does "backend" mean? Just the application layer? If you're writing a
Tomcat application, do you have to know how to debug Tomcat itself and submit
patches for issues? Do you need to know how to write a Java compiler or an OS
kernel? Do you need to know how to design microprocessors? What about how to
fab them?

Calling the top two layers of a very deep stack "full stack" is a bit strange.

