
Ask HN: What should a systems/low-level software engineer know? - avrmav
Tired of the hype on the web&#x2F;js (every week new library to learn about), I am thinking to switch career in the next 1-2 years, and move lower in the stack. I am into Rust in the last 3-4 months and I really enjoy it, but I would like to learn more, what would you consider essential books&#x2F;papers&#x2F;resources that a system&#x27;s programmer should know?
======
jandrewrogers
A systems programmer will specialize in subdomains much like any other
programmer, but there are some characteristic skills and knowledge that are
common across most of them. Systems programmer coding style is driven by
robustness, correctness, and performance to a much greater degree than higher
up the stack. Most systems programming jobs these days is C/C++ on Linux and
will be for the foreseeable future.

\- Know how your hardware actually works, especially CPU, storage, and
network. This provides the "first principles" whence all of the below are
derived and will allow you to reason about and predict system behavior without
writing a single line of code.

\- Understand the design of the operating system well enough to reimplement
and bypass key parts of it as needed. The requirements that would cause you to
to build, for example, a custom storage engine also means you aren't
mmap()-ing files, instead doing direct I/O with io_submit() on a file or raw
block device into a cache and I/O scheduler you designed and wrote. Study the
internals of existing systems code for examples of how things like this are
done, it is esoteric but not difficult to learn.

\- Locality-driven software design. Locality maximization, both spatial and
temporal, is the source of most performance in modern software. In systems
programming, this means you are always aware of what is currently in your
hardware caches and efficiently using that resource. As a corollary,
_compactness_ is consistently a key objective of your data structures to a
much greater extent than people think about it higher up the stack. One way
you can identify code written by systems programmer is the use of data
structures packed into bitfields.

\- Understand the difference between interrupt-driven and schedule-driven
computing models, the appropriate use cases for both, and how to safely design
code that necessarily mixes both e.g. multithreading and coroutines. This is
central to I/O handling: network is interrupt-driven and disk is schedule-
driven. Being explicit about where the boundaries are between these modes in
your software designs greatly simplifies reasoning about concurrency. Most
common concurrency primitives make assumptions about which model you are
using.

\- All systems are distributed systems. Part of the systems programmers job is
creating the illusion that this is not the case to the higher levels in the
stack, but a systems programmer unavoidably lives in this world even within a
single server. Knowing "latencies every programmer should know" is just a
starting point, it is also helpful to understand how hardware topology
interacts with the routing/messaging patterns to change latency -- tail
latencies are more important than median latencies.

The above is relatively abstract and conceptual but generalizes to all systems
programming. Domains of systems programming have deep and unique knowledge
that is specific to the specialty e.g. network protocol stacks, database
engines, high performance graphics, etc. Because the physics of hardware never
changes except in the details, the abstractions are relatively thin, and the
tool chains are necessarily conservative, systems programming skills age very
well. The emergence of modern C++ (e.g. C++17) has made systems programming
quite enjoyable.

Also, the best way to learn idiomatic systems programming is to read many
examples of high-quality systems code. You will see many excellent techniques
in the code that you've never seen before that are not documented in
book/paper form. I've been doing systems work for two decades and I _still_
run across interesting new idioms and techniques.

~~~
ken
> Know how your hardware actually works, especially CPU, storage, and network.

I learned about MIPS CPUs in college, 20 years ago, with Patterson and
Hennessy, and built one from logic gates. It was basically a high-powered 6502
with 30 extra accumulators, but split across the now-classic pipeline stages.

I understand that modern CPUs are much different than this. I understand that
there are deeper pipelines and branch predictors and superscalar execution and
register renaming and speculative execution (well, maybe a bit less than last
year) and microcode. Also, I imagine they're not built by people laying out
individual gates by hand. But since I can only interact with any of these
things indirectly, I have no basis for really understanding them.

How does anyone outside Intel learn about microcode?

~~~
jandrewrogers
For the average systems programmer, 90% of the really useful CPU information
can be found in two places that change infrequently:

\- Agner Fog's instruction tables for x86[1], which has the latency, pipeline,
and ALU concurrency information for a wide range of instructions on various
microarchitectures.

\- Brief microarchitecture overviews (such as this one for Skylake[2]), that
have block diagrams of how all the functional units, memory, and I/O for a CPU
are connected. These only change every few years and the changes are marginal
so it is easy to keep up.

Knowing the bandwidth (and number) of the connections between functional units
and the latency/concurrency of various operations allows you to develop a
pretty clear mental model of throughput limitations for a given bit of code.
People that have been doing this for a long time can look at a chunk of C code
and accurately estimate its real-world throughput without running it.

[1]
[https://www.agner.org/optimize/instruction_tables.pdf](https://www.agner.org/optimize/instruction_tables.pdf)

[2]
[https://en.wikichip.org/wiki/intel/microarchitectures/skylak...](https://en.wikichip.org/wiki/intel/microarchitectures/skylake_\(server\))

~~~
deepaksurti
Do we have similar resources for Discrete GPU's? I checked on wikichip, the
GPU information is hard to come by or I am missing some Google fu.

Thanks for sharing the above very useful resources, blocked my Saturday for
first scan!

------
osivertsson
From the top of my head, these are some areas I often would like other system
programmers to know better:

\- Understanding how to write purely event-based programs using e.g. epoll
since that will avoid lots of complexity and potential problems compared to
throwing threads at everything.

\- Floating point representation and gotchas ("What Every Computer Scientist
Should Know About Floating-Point").

\- Unit-testing as part of designing interfaces to get low coupling, sane
ergonomics, and forcing some thought to error handling instead of just working
in the happy-path case.

\- Interrupt- and thread-safety (races, mutexes, deadlocks, orderings etc.)

\- Storing state in maintainable ways (i.e. not just dumping C-structs to
disk).

\- Understanding basic computer architecture concepts like virtual memory,
caches, and pipelines.

\- Know what can be achieved with OpenMP and GPU programming using OpenCL/CUDA
etc.

\- Write great commit messages (What is the problem being solved, why this
way, etc.)

\- Basics of database and distributed systems theory like ACID properties and
Lamport timestamps.

\- Using regular expressions instead of lots of a complicated set of if-
statements.

\- Understanding networks regarding error sources and latencies.

~~~
jm__87
Maybe it is just me, but I've seen this issue multiple times where a
programmer used a regular expression thinking they were clever, only to have
it backfire later when there was some corner case not covered by their regex
(which likely would have obviously been caught if they had just taken the time
to write out each case as an if statement). You're usually just gaining
complexity in exchange for fewer lines of code, and I'm not sure that is
always the best tradeoff. Something to keep in mind when deciding whether or
not to use regular expressions.

With that being said, your list of things to know are all things I have to
know for my job (these are things almost all programmers should know, in fact)
and I would not consider myself a low-level or systems programmer, just an
application programmer.

~~~
Calloggs
Regardless of implementation (regex vs. conditionals), there should be
sufficient unittesting to make sure that all the corner cases are tested. For
embedded systems, the complexity tradeoff is relevant though, since a regex
library will (probably) take up more code space than some conditionals.

~~~
bunderbunder
In principle, I agree. In practice, it's easier to miss an edge or corner case
in a regular expression than it is in a series of conditionals. That's just
another consequence of the complexity tradeoff.

------
kahlonel
I'm an embedded software engineer. Here's my $0.02 on how to get started:

1\. Get a platform you can tinker around with, and recover quickly in case of
disaster. Raspberry pi is a good example.

2\. Learn how to find and read detailed register-level manuals of the
underlying chip you're working on.

3\. Learn what a programmer's memory-model is.

4\. Learn what peripherals are, and what they do. You may have to dig a bit
into basic electronics.

5\. Learn what user-space vs kernel-space is.

6\. Learn what device trees are, and how to prevent kernel from taking over a
peripheral.

7\. Based on above information, write a user-space driver for a simple
peripheral (e.g. GPIO or UART) in C (don't try to learn assembly, yet). You
will not be able to use the peripheral interrupts in user-space.

8\. Learn how to verify hardware functionality, and see your driver in action.
(Warning: high dopamine levels are reported at this point).

9\. Repeat steps 7 and 8 for all peripherals, and see how far you can go.

~~~
akiselev
_> 1\. Get a platform you can tinker around with, and recover quickly in case
of disaster. Raspberry pi is a good example._

I'd recommend something cheaper and simpler like the STM32F4 discovery boards.
You won't get linux and will have to program via JTAG but the documentation
for the STM32F4 is not as overwhelming as Broadcom's.

~~~
mbreedlove
Does knowledge of the STM32 family apply to other ARM chips?

~~~
roland35
I would say absolutely yes. I started with the STM32 parts and have then used
ARM Cortex devices from TI, Freescale, and Atmel. They all have different
peripherals and of course different peripheral libraries from the vendor but
overall they are conceptually similar. It can certainly be a pain in the neck
to switch a project from one device to another (lots of gotchas with pin
layout, peripheral minutiae etc etc) but starting a new project with a
different device should not be too hard!

Except for the Kinetis DMA, I never understood that one!!

~~~
sorisos
Spent a lot of time with the Kinetis DMA. Luckily I had nothing to compare it
with. Learned later it was not as other DMA:s.

------
Glawen
If by low level you mean embedded systems programming, you will definetly need
to be proficient in C. The other knowledge depends on the product that you
will make. For example I work in the automotive industry, where you need to be
interested in cars and how the different parts work. Knowledge of electrical
engineering and control theory is also valuable. There is a good book about it
if you wat to learn how cars works: Bosch Automotive Handbook.

If you go that path, bear in mind that you will not be involved in creating
clever algorithm. When safety is involved, you need to program very simple and
easy code. The complexity lies in how all the parts and ECUs are interacting
with each other. On the upside, you do not need to constantly learn new
languages and library, and you accumulate expert knowledge which is
interesting for companies.

~~~
kevmo
I don't do embedded systems, but as a self-respecting software engineer, I am
intending to develop proficiency in C/C++ over this next year. Does anybody
have any recommended books/courses/projects?

~~~
nwhm
I highly recommend Computer Systems: A Programmer's Perspective for learning
C. In particular chapter 3 of that book is what made it all click for me.
Understanding how C is translated down into assembly is incredibly useful for
understanding pointers and reasoning about your code.

EDIT. The book also comes with a collection of excellent projects, which the
authors have made freely available online at
[http://csapp.cs.cmu.edu/3e/labs.html](http://csapp.cs.cmu.edu/3e/labs.html)

~~~
jtsnow
Agreed! The labs are very informative and fun. One of my favorite undergrad
courses used this book/labs ~10 years ago. Glad to see it is still being used!

------
ninjakeyboard
I am somewhat of a systems engineer but more in the distributed systems space
rather than embedded systems. I guess:

\- runtime/space complexity and general algorithmic heuristics to know how the
system will perform, or to be able to fix scale issues in the code.

\- network characteristic (eg fallacies of distributed computing)

\- CAP theorem and applying it to real world designs

\- A co-ordination tool like zookeeper

\- Understanding of the technological lay of the land (eg compression like
snappy, different tools and approaches for communication like zeroMq, kafka

\- Metrics and alerting - eg get how to instrument with statsd and know how
and where to alert.

\- Concurrency concerns and scheduling - you can get away with understanding
futures and the actor model and can steer away from threading fortunately. (eg
elixir/OTP or Akka)

~~~
jschwartzi
It's important to realize that algorithms which scale very well on high
performance and distributed systems are frequently not the best algorithms to
use in an embedded system. For example scanning an array for each lookup
instead of using a hash is usually good enough when your array size is
relatively small(10 items to several-thousand). And it can mean the difference
between spending a week or two coding a hash map versus solving the really
important problem of building whatever product you need to build.

Embedded programming is all about forgetting all the really complicated
algorithms you might have learned in school, because they usually don't
matter, and when they do it's more important to be able to gather performance
data than it is to do something fancy.

Traditional concurrency is still a very real concern because DMA engines,
specialized coprocessors like the TI N2HET, and offload engines like modern
audio codecs all have their own internal firmware that your system must
interact with. Even getting the system to boot and to transition to low power
states requires you to understand clock trees, clock gating, power supply
states, and how to interact with any PMICs. Getting a real-time clock working
is a similar story.

Raspberry Pis are good for dipping your feet into a Linux system that uses a
specialized bootloader, but if you want to do anything truly embedded you're
going to have to go deeper than that and work with a system like the PIC32 or
even the RTUs on the Beaglebone Black.

If you're not using a JTAG at least occasionally then you're probably not
close enough to the hardware to be considered "embedded."

~~~
ninjakeyboard
For sure I'm not anywhere close to embedded.
[https://en.wikipedia.org/wiki/Systems_engineering](https://en.wikipedia.org/wiki/Systems_engineering)

------
antirez
There are good general rules in a few comments here, but I think it's ok to
say that in 2018 the specialization arrived to a point where system/low-level
is too large to reply in a good way? One thing is to write games engines,
another is to write networking code, yet another embedded systems, device
drivers and so forth. There are certain common aspects, but the details of
what you need to know change significantly.

~~~
senorsmile
This 1000x. The field is huge. Get the common basics down, then pick a
specialization that seems interesting or useful.

~~~
tracker1
On that, unless you _ABSOLUTELY_ love games and game engine design... the
field is relatively flooded and comparative pay may not be great. Not to
discourage, but making a decent living isn't a bad thing.

------
mlazos
I think for low level systems it’s important to be knowledgeable about the
design of OSes. For example, how memory works, caching and paging, how
programs are executed, how threads and processes are managed, how
communication with peripherals is typically done ... a good way of learning
this could be implementing a cpu emulator of some kind in rust, which will
probably give you a good idea of what areas you need to explore further. There
have been a few floating around on HN lately.

------
m00dy
I would like to start by Ulrich Drepper' famous paper What Every Programmer
Should Know About Memory [0]

[0]:
[https://people.freebsd.org/~lstewart/articles/cpumemory.pdf](https://people.freebsd.org/~lstewart/articles/cpumemory.pdf)

~~~
fizwhiz
Genuinely curious how useful this has been to you, personally.

~~~
m00dy
well, I'm the guy asking weird questions in interviews :)

~~~
59nadir
So it's only useful for figuring out what irrelevant question to ask people to
make them not want to work at your company? :D

Jokes aside, the guy asked how it's been useful and it's not terribly
convincing that your first (and only?) thought was interview questions.

------
turbinerneiter
If your are looking towards embedded:

* put a cap between power and ground

* get some nice ESD-birkenstocks

* don't listen to anybody telling you how it's done

The start of the art is BS and we need a revolution. There will be so many
people who tell you that you can only do it in C and if you don't, you can't
be taken seriously. The result is that everybody is keeping to C and nobody
invests time in bringing new ideas to the field. The chip vendors stubbornly
ship you really, really bad C SDK's and have no interest in doing better, for
reason beyond my comprehension.

As you mentioned Rust - there is a Rust working group for embedded targets and
they do cool stuff, but it's hard and a lot of work. Also, hardware is
basically a huge block of global mutable state, so that is a problem to wrap
ones head around.

But eventually we have to get rid of "C is the only serious option" which is
an argument made by people who know how to write "safe" C and perpetuated by
people who can't, but act like it.

[Before you react - I know this is an "extreme" statement and it's not 100%
accurate and there is much more nuance to it - it is exaggerated for comic
effect ;)]

~~~
Something1234
What would putting a single cap directly between power and ground be? Wouldn't
that be a short circuit or boom?

~~~
turbinerneiter
A filter cap. It is in parallel to the actual load and once it's charged,
there is not current anymore. It only booms when the voltage is to high.

Many chips, like a bluetooth chip, don't draw power continuously, but there
are spikes. These spike in current can lead to the voltage dropping to low. A
filter cap acts against that.

Sometimes, especially in prototyping circuits, where nobody took care of
properly designing a power supply, filter caps are the first cheap shot at
fixing weird behavior.

If you wanna know more, google the term, there is way better explanations then
mine, form people with way more profound knowledge.

------
n_t
What is your definition of system programmer? If you mean closer to hardware,
i’d recommend its worth the switch. Jobs are moving higher in stack and lower
layers are more and more stable and abstracted away. Most companies dont need
low layer developers off the shelf software is good enough for them. Exception
being embedded system teams or customized hardware (network, storage, server)
teams. Majority of things they do are age old techniques.

Only reason to go lower in stack is if you really ~like~ love hardware and
exploring nitty gritty of how things work. In that case, pick any OS book and
get ready to go down the rabbit hole :)

~~~
n_t
Not able to edit! But i meant NOT worth the switch.

~~~
collyw
Are the jobs still there or are they disappearing?

I am bored of the churn associated with stuff higher up the stack. I would
prefer to master a craft rather than chase new fads every few years.

~~~
aceperry
There aren't as many jobs in the embedded space as there are for higher level
software. Nowadays, lots of new embedded processors are hitting the market and
there will probably be a shake-out sooner or later. So the embedded space
isn't necessarily more stable than the web based stack. Lots of the processors
are getting huge with lots of memory, faster and more capable cpu's; all of
which will require programming at a higher level. Arduino's use C++, Raspberry
Pi's are usually programmed with Python (although you can use any language
that runs on linux), and most embedded processors can be programmed with the C
programming language. One niche that you might be interested in is the amount
of high level software that's required for today's newer and larger cpus/SOCs
that have a lot of connectivity, such as wifi, bluetooth, and software that
communicates with TCP/IP. Connectivity is driving a lot of the newer stuff.
with that, security is a big deal and might become required knowledge. There
are also a lot of experimental attempts at introducing new and higher level
languages on embedded systems. Rust, javascript, go-lang, elixir/erlang, are
some recent examples. The embedded space is beginning to experience the churn
that you're dreading in the high level stack.

~~~
Calloggs
These are valid points for embedded systems without latency/power
requirements, but I want to add that many embedded systems require rolling
custom assembly and writing everything else in C to avoid the overhead of
other languages and meet battery life and/or performance requirements.

~~~
alexdowad
As an extreme example: I just saw a new micro the other day which costs 3
cents per unit... but it only has 64 BYTES of RAM.

------
Iwan-Zotow
Some Ulrich Drepper papers:

\- on memory access/caches/memory hierarchy:
[https://www.akkadia.org/drepper/cpumemory.pdf](https://www.akkadia.org/drepper/cpumemory.pdf)

\- on DSO:
[https://www.akkadia.org/drepper/dsohowto.pdf](https://www.akkadia.org/drepper/dsohowto.pdf)

\- Agner Fog on CPU optimization, latencies, vectors etc:
[https://www.agner.org/optimize/](https://www.agner.org/optimize/)

------
sephoric
A bit on the more unusual side, one thing my son and I found very instructive
on "lower level" concepts is the fact that within PICO-8, which is already a
very user friendly high-level environment, you can use peek(), poke(),
memset(), memcpy and similar functions on any memory address that PICO-8 uses.
Literally all the capabilities that PICO-8 gives you, it exposes memory for it
and explains the conventions it uses to read/write that memory, so that you
could do it yourself. We were able to make a simplistic "paint" program that
draws a pre-drawn cursor sprite and changes pixels whenever you drag the
mouse, all using memset(), peek() and poke(), and avoiding line(), spr() and
doesn't use any Lua tables. It entirely does this by reading and writing
memory. It was a fun experience in learning certain useful low-level memory
concepts.

~~~
swolchok
> peek(), poke(), memset(), memcpy and similar functions on any memory address
> that PICO-8 uses

Surprised that this hasn't been exploited to build a C compiler targeting the
platform. (Unless I needed to Google something other than "pico-8 C
compiler"...)

~~~
sephoric
The only thing I can imagine that would be good for is learning how to write a
C compiler :)

~~~
swolchok
PICO-8 limits the textual size of your input Lua file, so I'd imagine that it
might be possible to squeeze more content into your game with a small
interpreter and packed binary format represented as a printable string.

------
mobilemidget
Know that third party libraries often are not a solution. Often have a limited
lifecycle, departed authors, and 90% of code you are not using, yet have to
consider in regards of security etc.

~~~
C1sc0cat
For what use? It doesn't make sense to write your own FFT library buy it from
NAG.

~~~
mobilemidget
You don't have to write everything yourself, but just saying be critical and
think ahead when choosing which libraries to use and how you use them. One
could even consider cherry picking (e.g. functions) from other
libraries/projects that you are allowed, licensed to do so to keep your
codebase as small as possible for various reasons, including security.

------
nuclx
Learn to get an intuition for how your code performs.

When is it appropriate to allocate memory from the heap? If you're in a
rendering or audio processing routine in realtime context, avoid it at all
cost.

Think about which parts of the code could profit from optimization. Does
learning an assembler for a specific platform pay off or can the compiler do a
sufficiently good job with -O3? Use profilers to identify performance
bottlenecks.

Think about portability. Does the code have to compile with ancient C89
compatible compilers [Mine has to, and I would be excited to see a Rust to C
transpiler]? Can you choose your compilers by yourself? Are they provided by
your customers?

------
lmilcin
Hi. Linux system / embedded developer here.

There are many different types of jobs. I will assume you mean low level
development on an otherwise normal operating system like Linux.

I would start by learning how Linux userspace is constructed. Dig up old Linux
From Scratch docs, build your minimal system, experiment with it. Try to
understand how different stuff is put together. How elf/static/dynamic
libraries works, etc. Look at your running processes (there shouldn't be many
if you do LFS) and try to explain everything you see (what this process does,
how it does its job, etc.)

Learn how kernel communicates with userspace. What are devices? What are
syscalls? What syscalls are available? How filesystems work? How devices work?
Etc.

Learn what is the job of Linux kernel. How memory management works, what is
virtual memory, what is the difference when programming in user/kernel space.

The best way to learn system programming is definitely not getting up to your
ears in a single open source project. You need variety of knowledge because
you want to understand in breadth how the system works. Once you get a job or
you figure out something interests you more there will be good time to
specialize (say you want to specialize in security or containerization etc.)

------
adrianN
Low-level software often moves quite slowly and backwards compatibility is
important. Embedded systems have to be maintained for decades. So knowing how
do handle legacy code, proper dependency management, release management, and
requirement management is quite important. Similarly the cost of bugs is
usually quite high, because updates are hard. So writing testable code,
testing and other QA activities are more important.

~~~
londons_explore
All of that stuff is changing though.

I would never ship a product today without remote signed upgrade ability.

All devices that plugs into a windows PC can use Windows Update for example to
upgrade firmware. Any device that sits at a remote location probably ought to
have a GSM modem in (adding only $1 to the cost of the device) for tracking
uptime and firmware updates. Any device which offers an API to other devices
should have 'provide firmware update' as part of that API.

Obviously security is a concern with updates, but by signing update files, and
making sure downgrades aren't allowed, the security benefits of being able to
patch vulnerabilities outweigh the disadvantages of the device manufacturer
being able to produce evil update files.

Those who say that your lightbulb/toaster/USB hub don't need automated
software upgrades are naive. Software on them will typically consist of many
libraries totaling perhaps hundreds of thousands of lines of code. Security
vulnerabilities will be found in that code, and even if the security of this
particular device isn't of concern, it can be used as a jumping off point to
attack other networked devices or for data exfiltration.

~~~
sokoloff
Where can I get a GSM modem and support components for $1 in BOM costs, let
alone $1 for finished product cost (inc intentional radiator cert and
distribution/channel markup)?

What about the $1-2/mo network charge? Or battery life considerations? I've
got sensors that run for 2-3 years on a CR123a battery. That power profile
isn't going to support a GSM connection.

~~~
londons_explore
Have a complete kit with free delivery for a dollar:

￡0.96 | DIY KIT GSM GPRS M590 gsm module Short Message Service SMS module for
project for Arduino remote sensing alarm
[https://s.click.aliexpress.com/e/b4tPrIy0](https://s.click.aliexpress.com/e/b4tPrIy0)

Lots of providers are happy to provide worldwide service for free, as long as
you pay $10 per gigabyte. I generally budget 1 check-in per day, of about 250
bytes, so a coin cell can easily power it for a few years with a total service
cost for 2 years of just a few cents.

------
k4ch0w
I would say learning a bit about the security implications involved. You don't
have certain things given to you like other languages.

You have to watch what compiler flags you use, like someone turning off stack
cookies, not using clang's sanitizers. Check out
[https://clang.llvm.org/docs/AddressSanitizer.html](https://clang.llvm.org/docs/AddressSanitizer.html)
it would have prevented the Heartbleed vulnerability if it existed at the
time.

You need proper bounds checking everywhere! You should fuzz your code with
something like AFL and if you don't have the time to setup test cases for it,
just send your program random junk and see if you can get it to seg. fault.

Multithreading is hard, and detecting multithreaded bugs is even harder.
Random monkey testing can sometime's help find these, but they are very hard
indeed. Monkey testing is just literally if a monkey was smashing your
keyboard and your program was open, what would happen?

Know the most vulnerable function calls by taking a look at banned.h from
Microsoft's SDL. It looks like it's no longer on the official website, but the
author put it up here Take a look at
[https://github.com/x509cert/banned/blob/master/banned.h](https://github.com/x509cert/banned/blob/master/banned.h).
Sometimes you can't avoid using these functions but know why they can be
considered bad.

A proper Makefile is your team's best friend. The same can be said with one
build machine for your whole team. It is easier now than ever to build and
distribute it between your team with Docker. This is just a personal opinion,
but I think downloading all the exact library versions you need once and
putting them in a Docker image will save your team some pain in the future.

~~~
sebcat
> You have to watch what compiler flags you use, like someone turning off
> stack cookies, not using clang's sanitizers. Check out
> [https://clang.llvm.org/docs/AddressSanitizer.html](https://clang.llvm.org/docs/AddressSanitizer.html)
> it would have prevented the Heartbleed vulnerability if it existed at the
> time.

If that was true, then so would Valgrind have, and Valgrind was in wide use at
the time.

It was however more complicated than that. If my memory serves me right -
OpenSSL had its own memory management.

------
drablyechoes
Topics to study should include: Unix IPC, signals and syscalls, filesystem and
kernel implementation details, and network/socket programming.

Being able to read and write C programs goes a really long way.

Beej's guides are a good resource. I learned a lot from them.

Guide to network programming:
[https://beej.us/guide/bgnet/html/multi/index.html](https://beej.us/guide/bgnet/html/multi/index.html)

Guide to Unix IPC:
[https://beej.us/guide/bgipc/](https://beej.us/guide/bgipc/)

The Unix Programming Environment (Kernighan & Pike) is also very good.

[http://files.catwell.info/misc/mirror/the-unix-
programming-e...](http://files.catwell.info/misc/mirror/the-unix-programming-
environment-kernighan-pike.pdf)

A good exercise would be to write a simple IRC-like chat server and client in
C.

------
roland35
One important thing for an embedded engineer (just like any software
engineer!) to know is how to select an appropriate solution for a given
problem. This includes both hardware and software, and do it yourself vs off
the shelf.

I think it can be summed up with some questions to ask at different points in
a project:

\- Should I use a microcontroller or a processor? If a microcontroller should
I use a simple 8 bit or more featured 32 bit?

\- Do I need an operating system like Linux, RTOS like FreeRTOS, or bare
metal?

\- Are there existing code modules out there to help kick start the project?
Like SD card libraries, ethernet middleware, etc

\- Should I design a custom PCB or look at development kits, or off the shelf
electronics?

\- Where should I design in flexibility in the project? What requirements can
be solidified to simplify the design?

Knowing about what is out there helps pick appropriate solutions to problems,
which will save the most time in the future.

~~~
jamesadevine
In your opinion is 8-bit even a realistic choice anymore? Many ARM CPUs offer
way more bang for your buck nowadays (unless i'm looking in the wrong place).

~~~
AnimalMuppet
What do you want? Do you want a system to play with for your own learning? Or
do you want to build a shipping product?

If for your own learning, do you want a full-powered environment? Or do you
want a simple system that you can learn all of, even if it's more of a toy?

If it's a shipping product, do you care more about ease of development, or
about total parts cost? (The difference is often quantity that you expect to
ship - 10 cents in part costs matters if you expect to make 100 million of
them.)

------
flipnotic
In embedded and systems-level programming, something I find indispensable is
knowing what values fit in what types (for example, an 8-bit type can
represent a maximum of 256 values; a 16-bit type can represent 65536 values).

This comes up incredibly frequently, and having it be second nature will
benefit you.

A lot of the time, this gives you a starting place for how your inputs and
outputs should look, what your method signatures should be. For example, if
your data can can fit in a uint8_t, why waste bytes on a uint32_t?

This isn't the be all and end all, especially since modern architectures use
larger word sizes/RAM is plentiful/bandwidth is cheap, but it can be a helpful
way to frame the conversation when you're architecting software (especially
network protocols).

~~~
m-ee
Also the ability to understand and check the performance impact of these
choices. For example at a old job on an embedded motor controller the previous
engineer had learned 'floats are slow never use them on a microcontroller' and
went and removed them from the code wherever he found them. However in the
motor control update loop this meant using longs (or possibly even long longs,
I don't remember) in a couple calculations. A quick check with a pin toggle
showed this was significantly slower than just using floats.

I'm an ME who went into embedded systems, so I'm not sure what qualifies as
low level for a web developer and if checking timing using an oscilloscope or
saleae is feasible for the type of work OP wants to do. But even just looking
at the assembly would have made things obvious in my example.

EDIT- This made me realize that a rudimentary understanding of the assembly
language for your architecture will be very valuable. Doesn't have to be
enough to actually write any code in it, but it's great to be able to take a
look at what was generated when things are acting weird.

~~~
zozbot123
Fixpoint arithmetic is going to be significantly more efficient than floating
point in many cases, except where you need to use relatively-uncommon
operations/functions (which are hardware-accelerated in the FPU), or in cases
where the same value might span multiple orders of magnitude (i.e. a direct
win for the floating point representation). The inner loop of a
microcontroller is not generally one of these cases, though exceptions might
be out there.

~~~
m-ee
I was surprised it was faster, from what I remember to avoid floating point
operations several large values had to be multiplied together then divided
which required using a long long to avoid overflow. The division operation was
quite slow. Thinking back I'm not 100% sure it was in the inner control loop
but at the very least it was required to generate the new motor position.

Anyway a bit toggle tells all, and if you're making speed optimizations you
should be checking that things actually got faster.

------
pizlonator
My advice:

\- become good at either C or C++ or both. Lots of fun jobs/projects involve
codebases that happen to be written in those languages.

\- become fearless and systematic with assembly. You don’t have to be great at
it. You just have to have had enough experience looking at it and writing it
that you can hack it if you have to.

\- learn to read code quickly and accurately. The only way to do that is a lot
of practice. It’ll be hard at first but as you practice your reading speed
will go up by >10x and eventually it’ll feel like second nature.

\- become great at working with large code bases. What makes “systems code” so
interesting is really just how big “systems” are. This kinda goes along with
the bit about reading code - reading is how you survive in large code.

------
CodeWriter23
A topic not covered thus far: what happens when the computer is powered up?
Dig into writing your own boot ROM/FLASH. Nothing like having to research the
various CPU registers that require initialization during boot to give you an
intimate understanding of what the core of the system is.

A project I worked on that was also an interesting challenge was a RAMless
monitor ROM. Write a program that will boot up the monitor (not the OS) and
provide an interactive console that uses no RAM at all. Commands are to poke
or dump RAM, read/write IO ports, probe the bus for devices etc. You can’t use
the CALL instruction because you have no stack. Test your code by pulling the
RAM modules.

------
krosaen
Brendan Gregg's website has a lot of useful content on system performance
analysis

[http://www.brendangregg.com/](http://www.brendangregg.com/)

------
ensiferum
System isn't necessarily low-level. System code is and can be quite high
level, the distinction to just calling it "application" or "application level
code" is that it caters mostly to system (no direct user interface) or
applications (such as system libraries) instead of end users.

Then you can have "low level" as in close to the metal, for example drivers
that talk to a hardware. That being said even these are (and should be written
mostly) at fairly high level (especially in the user space) and it's only a
relatively small part that has to mess with HW specifics.

And then of course there's some embedded IoT stuff that has also different
layers of software although for some people working anywhere in the embedded
stack would qualify as "low level".

Generally speaking you'll want to lean on a few core technologies and APIs
such as POSIX (on linux), have a decent understanding of the von neuman
architecture and a solid grasp of C/C++. Having a decent idea of how different
subsystems such as network or TCP/IP stack works and such can be useful
depending on the domain.

~~~
lsofzz
> System isn't necessarily low-level. System code is and can be quite high
> level, the distinction to just calling it "application" or "application
> level code" is that it caters mostly to system (no direct user interface) or
> applications (such as system libraries) instead of end users.

That's exactly my point as well. Well said - ha at least better than I would
be able to explain myself.

------
notacoward
I'd start with everything by Hennessy and Patterson. That should cover most of
how CPUs and memory systems work.

I grew up with Tannenbaum for networking, though most people went toward Comer
and Stevens. Maybe there's something even more current.

I honestly don't know anything as good for storage, which is funny since it's
my own specialty. I can try to cobble together a reading list if you'd like.

Something on operating systems. Vahalia for an overview, Bach for System V,
gang of four for BSD. Yes, study something besides Linux ... but do study
Linux as well. Not sure which books are good for that.

Something on compilers. Is the dragon book still the go-to reference here?

Even if you don't work in those specific areas, that should give you the
grounding to study more. I'd also throw in something on databases, but don't
know a specific book/author. Distributed systems much like storage: definitely
learn, can't think of a good single source, can create a list if you want (I
think I did one for a colleague not too long ago).

Good luck!

~~~
rdc12
I would be keen to see the list on both storage and distributed systems.

For compilers the dragon book has fallen out of favor (especially as a first
compiler book/source). Modern Compiler in ML (don't touch the Java or C
versions) and Engineering a Compiler tend to be the goto books now.

------
csours
Signal Analysis, Electrical Engineering, Failure Mode Analysis.

The things you talk to will fail, you should be prepared to deal with those
failures.

~~~
robertAngst
After doing embedded work, YES! THIS!

Electrical Engineering is basically being able to read drawings and choose
components for your application.

Signals I found fun, but I can imagine doing a digital application and this
being difficult.

FMEA comes with the job IMO, but a good tool to have.

------
lmedinas
Learn how to use Debugging Tools and understand how to system works (Kernel,
libraries, application, C programming) etc...

This is the best advice.

------
protomok
My two cents - keep the books/papers/resources on the shelf and go out and
build an embedded project. Sure, you could read about interrupts, registers
maps, chip datasheets, etc. but the rate of learning will be much greater if
you learn by building.

For example, why not build a home automation system?

Start with a Raspberry pi and start hooking up peripherals...light switchtes,
timers, sensors, etc. You'll stumble across buses like I2C/SPI, you'll learn
about networking. You'll figure out what registers are and learn what
interrupts are and what they mean.

You'll get lazy rewriting communication code and stumble across messaging
frameworks like MQTT to communicate with devices on your network. You'll run
out of money using Raspberry Pis for each new device you build, and you'll
find cheaper ways of doing things like desiging devices using MSP430 or
ESP8266s.

You'll make mistakes, and you will learn. Best of luck on this new adventure!

------
chubot
Designing and writing explicit state machines. They seem to come up more in
low-level code than high-level code.

I prefer to generate them, so I would add to that "tools for generating C code
and state machines".

(Tangent: A guy who worked on Verilog and FPGAs told me that the "whole idea"
is state machines... and they are hard to get right.)

~~~
ginsmar
There is a C framework based in state machine:
[https://bitbucket.org/yuneta/](https://bitbucket.org/yuneta/)

------
edsiper2
Since your goal is to have a career switch I would recommend you the following
approach:

\- Essential books and papers are good as a reference to learn and improve but
focus first on hands-on stuff.

\- Get involved in an open source project that fits the 'system level'
category, learn how it works and try to contribute to it (again, put your
hands-on)

\- Start with development in C language, a good tool to build would be a TCP
client and server (kind of 'echo' program). The goal of this will be to learn:
how to compile a C program, basics of memory management (malloc, free,
realloc), networking (create a socket, bind, listen, connect), transfer data
(write, read) and so on.

\- If possible, learn using a native Linux environment.

All of the suggestions above will pay off if you are persistent and keep
trying learning by building; of course, getting involved in an open source
project is a must if you want to have a successful career.

------
yitchelle
Assuming that you want to move into embedded.

Focus on C and follow up with an understanding of the underlying assembler
code. Haven't played with Rust for about 1 year, so I don't how how mature it
is these days.

Also have an idea of how to debug the code. Debugging on embedded is very
different to debugging a web/js app.

~~~
ceidsness
Linux Device Drivers 3rd Edition is a bit dated but is still a wonderful
introduction to systems programming in C. It's for Linux, but it presents
information that is useful when working with any system.

Low-level systems programming is hard and can be a bit boring at times. I
agree that debugging is very different from debugging on higher level systems.
Most of the time an oscilloscope or an LED is the best debugging tool. Also,
in well written low level code you tend to have more code making sure that
everything is okay than you have actually doing stuff, which can be tedious.
It takes a certain kind of personality to enjoy it.

~~~
lsofzz
> Linux Device Drivers 3rd Edition is a bit dated but is still a wonderful
> introduction to systems programming in C. It's for Linux, but it presents
> information that is useful when working with any system.

[edit] I have the LDD and [https://www.amazon.com/Linux-Kernel-Development-
Robert-Love](https://www.amazon.com/Linux-Kernel-Development-Robert-Love)
book. I got confused about the authors name :)

The author Robert Love has done a tremendous job on _Linux Kernel Development_
book. Every aspiring "In-betweeners and Low-Low Level engineers" [0] should
read this book. It's awwwwespiring.

And please read LDD too!

[0]
[https://news.ycombinator.com/item?id=18882486](https://news.ycombinator.com/item?id=18882486)

------
afandian
Write a kernel extension. The Mac OS documentation is very comprehensive and
helpful. It's a different type of programming to most people's day-to-day
stuff. And it's very rewarding to see it load up into the OS (or crash the
whole system as the case may be).

------
jupiter90000
Here is a good article for you to read written by a long time embedded systems
guy directed at those looking to get into the industry:

[http://www.ganssle.com/startinges.htm](http://www.ganssle.com/startinges.htm)

------
epiczz
1\. Learn basic C and C++ programming, which is always a good starting point
for other higher and lower level programming languages

2\. Learn about data structures and memory handling (pointers)
[https://www.geeksforgeeks.org/data-
structures/](https://www.geeksforgeeks.org/data-structures/)

3\. Try to understand how an OS works (e.g. linux)
[https://github.com/tuhdo/os01](https://github.com/tuhdo/os01)

4\. Buy an Arduino or alternative with some I/O.
[https://www.arduino.cc/](https://www.arduino.cc/)

------
rhexs
Last I checked, most embedded interviews (presumably barring Google etc.) can
still be passed if you work through most of K&R (which is quite easy and
accessible), know a bit about OS's, and memorize some trivia about locks.
Worth reading about some of the deep C nuances as well if your interviewer
frequents ##C on freenode.

Linked lists, bitcounting problems (which is covered in the first few chapters
of K&R), etc. are all still popular.

The biggest issue you might have is the difference in salary and available
positions you're going to see going from FANG Backend Dev at 5 bazillion a
year or whatever it is now to 80-120K.

------
mathieubordere
A great resource to dive a bit deeper into the stack is
[https://www.amazon.com/Computer-Systems-Programmers-
Perspect...](https://www.amazon.com/Computer-Systems-Programmers-
Perspective-3rd/dp/013409266X) and the accompanying labs found at
[http://csapp.cs.cmu.edu/3e/labs.html](http://csapp.cs.cmu.edu/3e/labs.html).

Just make sure you don't buy the `international edition` of the book, it's
unreadable due to the amount of errors in the exercises.

------
O_H_E
Take a look at

Nand to Tetris:
[https://news.ycombinator.com/item?id=18519883](https://news.ycombinator.com/item?id=18519883)

Nand to Tetris II:
[https://news.ycombinator.com/item?id=14526344](https://news.ycombinator.com/item?id=14526344)

I asked a question in a here before that might have some relevant answers:
[https://news.ycombinator.com/item?id=17282466](https://news.ycombinator.com/item?id=17282466)
(not exact topic)

------
sbr464
I was also curious about choosing or creating a file format, when creating a
low level program that outputs a lot of data.

Any recommendations or popular reading?

I realize some choices are for proprietary reasons, then compression, but that
aside, there seems to be thousands of choices available.

Compared to the web, using json etc, and releasing a schema, but not
necessarily creating a new file extension.

~~~
janders
My current go to for this is using SQLite. It's basically made for this
purpose. If that doesn't serve, I like the idea of Apache Avro, but some of
it's C++ bindings are a little lacking in my opinion.

~~~
StillBored
This is a fantastic first choice, particularly as it sets you up for using a
more "real" database for sharing data/scaling in the future.

OTOH, you have to know when no to use it and step up (down?) to something that
is text editor hack-able (XML!?) or has barn burner I/O abilities (yah
actually just dumping raw buffers with regularized binary data to disk). Or
for that matter is used to exchange data with other apps with other services
(JSON, and the long list of other data dependent formats, although for at rest
exchange I have to point at XML again).

------
__d
You should learn C. Ideally, you could learn C++ as well, but I think it's
best to learn C first. And then treat C++ as a completely separate language.
Rust is a good thing too, but it's a little too soon for it to be broadly
useful.

Then you should learn Unix. From an understanding point of view, I think it's
probably better to learn something like FreeBSD, NetBSD, Xv6 ... Linux is very
pragmatic, and very general, and so it doesn't have the purity that smaller,
more focused or curated systems have. Once you have a handle on Unix, look at
other OSes: Plan9, Minix, FreeRTOS, L4, etc.

Then networking: I suggest starting with IP, TCP, ARP; then step down to the
physical layer: Ethernet, Token Ring, 802.11, hubs, and switches; then static
routing, RIP, OSPF, and BGP; maybe look at mesh routing. Then some application
layer stuff: DNS, NTP, SSH, NFS, LDAP, HTTP, etc. Reading the RFCs is really
valuable, and they're remarkably accessible up until say 2500 or 3000 or so.

Security: symmetric and asymmetric crypto, Kerberos, SSL, SSH, OAuth, etc.
Read up on pen testing, social engineering, defensive programmings, fuzzing,
etc.

Databases: both relational and otherwise. SQL. Wrap your head around how a
filesystem and a database are the same and how they're different.

Messaging: some sub-set of protocol buffers, Cap'n'Proto, Avro, Thrift, XDR;
brokers vs. p2p; pub-sub vs. directed. There are hundreds of system you can
look at here: pick a few that look different.

Learn about complexity analysis, distributed consensus, locking, concurrency
and threads.

so far as tools go, you need to understand a debugger (how to use it, and how
it works), packet capture and analysis (Wireshark is good), profiling and
performance analysis.

That's probably a decent coverage for the software side. The exact focus will
differ depending on embedded/real-time vs enterprise, etc.

From the hardware side, I think it's worth starting with an 80's or earlier, 8
or 16-bit system, and learning how it works to the digital logic level. What a
simple microprocessor actually does: fetching, decoding, execution units, etc.
A Z80 or 6502 or similar system is a pretty simple circuit, and it's worth
really grokking how it works.

From there, you can move forward to more complex CPUs, newer memory
architectures, newer buses, etc. But it's _much_ harder to dive straight into
a modern x86 or ARM CPU and try to understand how it works.

It's a this point that reading Drepper's memory article, and the "everything
you should know about latency" article(s), etc, really start to be useful,
because you've got a solid grounding in what's underneath them.

You don't need to do this all at once, or before you start working more on
backend or systems level code: I'd guess it took me close to 10 years to feel
like I had a decent grasp on most of it.

~~~
lsofzz
Nice points. I've prepared what I believe is a good non-exhaustive list
without getting very specific. I could be wrong ..

\- Embarrassingly low-level (chip developers): OS, C/VHDL/Verilog, MA, DS,
GPH, EE, Apathy, SE

\- The in-betweeners (kernel, device, storage developers): OS, C, MA, DS, GPH,
Apathy, Crypto, SE

\- Low-Low-level systems: OS, C, DS, GPH, Apathy, Crypto, SE, FTDS, MSG, NW.

\- Low-High-level systems: C/Rust, DS, GPH, Apathy, HDS, Crypto, SE, FTDS,
MSG, NW, TL.

\- Unicorn _rock-star_ systems hacker (perhaps only a handful of these
creatures exists :-): Multiple OS, C/Rust/JVM/Erlang/Haskell, Multiple MA, DS,
GPH, EE, Crypto, FTDS, MSG, NW, HDS, SSE, TL.

Abbreviations used:

GPH = Good Programming Hygiene

EE = Electronics and Electrical Engineering

OS = Operating System

MA = Modern Assembly

DS = Data Structure

FTDS = Fault-Tolerant Distributed System

MSG = Messaging System

NW = Networking

TL = Toolings (the darlings of UNIX)

HDS = High-Density Systems

SE = Security Engineering

------
DerDangDerDang
Watch Mike Acton’s CppCon 2014 talk on Data Oriented Design. His advice may
not be at all applicable to whatever field you end up in (and it’s a
deliberately contentious talk) but IMO it’s essential for a systems level
programmer to have enough understanding of the related concepts to be able to
argue about them convincingly.

------
vectorEQ
some resources i find interesting and helpful learning low level system stuff:
[https://wiki.osdev.org/Expanded_Main_Page](https://wiki.osdev.org/Expanded_Main_Page)
[http://www.osdever.net/](http://www.osdever.net/)

That and documentation for the platform you are working on, in my case
intel/amd manuals but not sure what system you are working on, i presume one
of those though.

it's also interesting to look into assembly/ assemblers, disassemblers,
perhaps look at capstone / keystone frameworks to learn about that. linkers &
loaders is also interesting.

it really depends on where you want to go with low level/ system things what
is relevant though, as a lot of it involves platform specific details.

------
cozzyd
You should know C like the back of your hand and at the very least how to read
compiler-generated assembly.

------
jeffreyrogers
The other posts are good, but I will give you the advice someone gave me (that
I ignored) when I was considering the same thing: Consider doing it as a
hobby. In general web stuff pays better. This seems crazy to me, but that's
where the demand is.

------
Nursie
If you want to start looking at OS level stuff, if you ever want to do
embedded work...

C

You may not need to do new work in it, but the large body of existing stuff is
in C and that's not changing for a while.

If you want to move into the mid-stack then Java is pretty good these days,
python too.

------
dabockster
Become an expert on memory safety. All it takes is one unchecked array bound
for a program to get pwned. And don't rely on ASLR since that can get pwned as
well.

If you don't know what ASLR is, you're not ready for systems programming.

------
valarauca1
For High Throughput compute, and cache correctness. Here are the primers I can
give. None of these are required. Strong langauge, cs, interpersonal skills,
team inter-interoperability, and commitment to quality (via unit and
integration testing, and skill working with liners) can get you a lot further.

The resources I'm linking are supplementary to the above, and you'll likely
encounter them in the wild. But they'll help you build a base of knowledge,
and give you terminology to search for, and work with.

\- What Every Programmer should know about memory:
[https://people.freebsd.org/~lstewart/articles/cpumemory.pdf](https://people.freebsd.org/~lstewart/articles/cpumemory.pdf)

\- Cache Obviousness:
[https://www.youtube.com/watch?v=bY8f4DSkQ6M](https://www.youtube.com/watch?v=bY8f4DSkQ6M)
(the suggestions, and terminology are important)

\- Parallelism: (This is a good primer, there are a lot of complementary posts
linked on the site) [https://preshing.com/20120612/an-introduction-to-lock-
free-p...](https://preshing.com/20120612/an-introduction-to-lock-free-
programming/) [https://preshing.com/20120913/acquire-and-release-
semantics/](https://preshing.com/20120913/acquire-and-release-semantics/)

\- If you plan on working with linux these is an excellent reference:
[http://man7.org/linux/man-
pages/dir_section_2.html](http://man7.org/linux/man-pages/dir_section_2.html)
remember there is no magic in Linux, everything eventually has to go through a
system call. So if you learn the systemcalls, you can can learn how things
work :)

\- Fog's optimization resources are awesome:
[https://www.agner.org/optimize/](https://www.agner.org/optimize/)

\- MIT courseware:
[https://www.youtube.com/watch?v=ytpJdnlu9ug&list=PLUl4u3cNGP...](https://www.youtube.com/watch?v=ytpJdnlu9ug&list=PLUl4u3cNGP63WbdFxL8giv4yhgdMGaZNA)
[https://www.youtube.com/watch?v=HtSuA80QTyo&list=PLUl4u3cNGP...](https://www.youtube.com/watch?v=HtSuA80QTyo&list=PLUl4u3cNGP61Oq3tWYp6V_F-5jb5L2iHb)

\- This cheat sheet is worth committing to memory:
[http://www.bigocheatsheet.com/](http://www.bigocheatsheet.com/) the reference
links are also great for building up knowledge

\- I highly recommend CMU DB open course ware: (intro)
[https://www.youtube.com/watch?v=vyVGm_2iFwU&list=PLSE8ODhjZX...](https://www.youtube.com/watch?v=vyVGm_2iFwU&list=PLSE8ODhjZXja3hgmuwhf89qboV1kOxMx7)
(advanced)
[https://www.youtube.com/watch?v=poEfLYH9W2M&list=PLSE8ODhjZX...](https://www.youtube.com/watch?v=poEfLYH9W2M&list=PLSE8ODhjZXjYplQRUlrgQKwIAV3es0U6t)

This should give you a good primer on Concurrency, and DB's. For networking
likely a basic TLA+ certification class will be 99% review, but for the things
it isn't will offer great insight.

~~~
sbmthakur
That's an interesting list. Thanks for compiling.

------
ajmarsh
This seems like a good place to start.

[https://github.com/kamranahmedse/developer-
roadmap](https://github.com/kamranahmedse/developer-roadmap)

------
mathieubordere
And if you're going to write multi-threaded programs or libraries that are
going to be called by multiple threads, then please learn how to write decent
thread-safe code.

------
tomer_shalev
[https://github.com/nhivp/Awesome-Embedded](https://github.com/nhivp/Awesome-
Embedded)

------
samfisher83
Embedded programing pay sucks compared to the web. I wouldn't do it. Just look
at the number of web jobs vs system programmer jobs.

~~~
icedchai
I've done embedded a few times in my career (usually higher level, embedded
Linux) though I also did some stuff with PIC microcontrollers for a bit.

The pay sucks, but it is _much_ more enjoyable work than web development.
There's a lot less BS to deal with.

~~~
samfisher83
I feel there is the BS there too. You need to get board support packages.
Sometimes the memory map is wrong. If you can get board up it might be a
timing issue and you have to solder onto the Trace etc.

~~~
icedchai
It also feels more like "real" programming. A lot of web development is
dealing with BS problems resulting from building apps on a platform that was
never meant for applications in the first place.

------
blizzard8
Take CS-61 taught by Eddie Kohler at Harvard

~~~
anonuser123456
I don't know what's in the CS-61 course, but his UCLA-CS235 Advanced Operating
Systems course was REALLY useful. You build the important bits of a
multitasking OS.

[http://read.seas.harvard.edu/~kohler/class/10f-aos/](http://read.seas.harvard.edu/~kohler/class/10f-aos/)

------
krtzer
The word "system" is used a lot to describe many different collections. For
example, the molecules that act on glucose to create ATP is considered a
"system," as in something like an RF chip that has Tx/Rx on the same piece of
silica. Can someone help provide a definition for what a system means in the
Hacker News context? What is "low-level" as well.

For me, low-level is assembly, not C.

~~~
dorfsmay
Does anyone beside compiler writer write in assembly? For what systems? Why
wouldn't they use a compiled language?

~~~
AnimalMuppet
When trying to solve the worst bug I ever encountered, the critical clue came
when I looked at the assembly output that the compiler produced.

You don't have to write it (much). You should be able to read it (with the
instruction set manual if needed), and maybe be able to write a little bit of
it in a pinch.

------
k0t0n0
if you cannot find the right answer here you can try rust subreddit (r/rust/).

------
rramadass
"Systems Programming" at the minimum, requires knowledge of Language
internals, Runtimes, Compiler Toolchains, OS internals, Drivers, Computer
Architecture/Organization, CPU/Multiprocessors and Networking. There are other
domains like Security, Parallel/Distributed Processing etc which are cross-
functional or orthogonal to the above and can be focused on as needed.

But be warned, earning potential-wise you may lose out(unless you get lucky)
to the latest whiz-bang Web/Mobile technology/framework brouhaha. That is the
nature of the market. However i consider that the satisfaction of learning and
"knowing" how things work, more than makes up for the small loss in earning
potential. This technology is also more fundamental and stable and thus will
not go away anytime soon.

I have found the following papers/books (somewhat different from the most
commonly cited) useful in my study;

 _Languages:_ Fluency in C is a must. It is THE "portable assembly" language
and is available on everything from 8-bit MCUs to multicore servers. You can
also link C modules to everything under the sun thus allowing you to extend
almost any other language. C++ is also needed; but judicious usage as a
"better C" rather than heavy-duty OOP/Generic/Template-mataprogramming
madness.

\- Computer Systems: A Programmer's Perspective 3rd ed. based on x86-64. You
might also want to get the 1st ed. which is based on 32-bit x86. These books
cover the HW/SW interface and thus includes almost all the topics under
"Systems Programming".

\- The C Companion by Allen Holub.

\- Inside the C++ object model by Stan Lippman.

 _Tools /Toolchain:_ Knowledge of the GCC toolchain is a must.

\- The Definitive guide to GCC by Hagen

\- ELF: A Programmer's perspective; paper by H.J.Liu

\- C++ Under the hood; MSDN article by Jan Gray

\- The Practice of Programming by Kernighan and Pike.

 _OS /Kernel/Drivers:_

\- UNIX Systems for Modern Architectures: Symmetric Multiprocessing and
Caching for Kernel Programmers by Curt Schimmel.

\- Linux Kernel Development 3rd ed. by Robert Love.

\- Essential Linux Device Drivers by Venkateswaran.

\- Embedded Linux Primer 2nd ed. by Hallinan

\- The Unix Programming Environment by Kernighan and Pike

\- Advanced Unix Programming 2nd ed. by Rochkind

 _Computer Architecture /Organization_

\- Modern Processor Design: Fundamentals of Superscalear Processors by Shen
and Lipasti

\- Computer System Design: System-on-Chip by Flynn and Luk

 _Parallel /Distributed processing:_

\- Foundations of Multithreaded, Parallel, and Distributed Programming by
Andrews

\- The Art of Multiprocessor Programming by Herlihy and Shavit.

 _Bare-metal Embedded:_ Where the "rubber meets the road"

\- Embedded Systems Architecture: Explore architectural concepts, pragmatic
design patterns, and best practices to produce robust systems by Lacamera

\- Patterns for Time-Triggered Embedded Systems by Pont.

------
snotrockets
The cost of everything, and the value of nothing.

------
goldenkey
Smashing the Stack for Fun and Profit :-)

------
pnathan
a lot of long and good answers.

but, a traditional trial by fire is to write an operating system.

------
evandrix
assembly

------
mesaframe
teachyourselfcs.com

------
aviv
That while they are grinding their professional life on Kernel development to
barely make ends meet, others with Geocities-level HTML skills are making 10
times more money with Clickfunnels selling crap nobody needs.

------
the_other_guy
try to make a docker toy alternative in C or Rust, you will learn a lot

------
amelius
Any book recommendations?

