
Low-Level Programming University – A roadmap to becoming a low-level programmer - znpy
https://github.com/gurugio/lowlevelprogramming-university
======
CalChris
Don't go there. There's really not a lot of work. But if you must, low level
is more a calling than a learned skill.

You probably shouldn't be learning assembler. First, compilers are really
quite good. Yes, it's possible to beat them (I do) but generally not by much.
And not by much ain't gonna put bacon on the table. You can probably get what
you need from gcc inline asm() calls. Take a look at the linux sources and
figure out why and when assembly is used there:

[http://stackoverflow.com/questions/22122887/linux-kernel-
ass...](http://stackoverflow.com/questions/22122887/linux-kernel-assembly-and-
logic)

Secondly, writing in assembler is _not_ low level. You just think it is. You
should really be understanding caches and you can improve your cache
performance in C.

Anyways, unless you deeply know what's going on inside of the
microarchitecture of a modern superscalar, out of order, speculative,
renaming, μop-cached, hyper-threaded, multicore beast then you shouldn't be
fooling yourself by writing in assembler.

[http://blog.erratasec.com/2015/03/x86-is-high-level-
language...](http://blog.erratasec.com/2015/03/x86-is-high-level-
language.html)

Unless you've really read Intel's _Intel 64 and IA-32 Architectures
Optimization Reference Manual_ (and ARM's _Cortex®-A72 Software Optimization
Guide_ ) and meditated on the suras of Agner Fog's _Microarchitecture_ you
won't even know what's going on with something as simple as _mov RAX, RBX_.

Look, most compiler writers don't even know this stuff (Intel C Compiler yes,
llvm occasionally) and frankly, it isn't very useful because Intel spends a
billion dollars a year to make your bad x86 code run reasonably fast. Consider
a switch statement which compiles into an indirect branch, _jmp reg_. That
branch has to be predicted by the BTB _before the jmp reg instruction is even
fetched_ and that's really hard to do. Every year they get better and better
to the point that you're not even aware of it. But if you want to the help the
CPU out, you could put a UD2 right after the _jmp reg_. This is insanely hard
to understand and will help very little.

Don't go there.

~~~
briansteffens
I agree not to go into low level programming expecting a wealth of job
opportunities to suddenly open up, but I wouldn't tell people not to go there
at all. Not only is assembly really fun to play around with, I feel like I've
gotten a lot out of the bits of assembly I've read/written. Even though I've
never written assembly code for work, being able to read the disassembly in
gdb has come in handy before. Also, a lot of the subtleties of C/C++ never
quite clicked for me until I had an idea of how the generated assembly would
work.

It's similar to learning a functional language. I have no idea if I'll ever
use Haskell professionally but learning a bit of it has been a good way to see
problems and logic differently. I think both assembly and Haskell have made me
a better programmer, even if I never become truly proficient in either or use
them directly in my job.

~~~
pacman128
Totally agree. When I taught x86 assembly in college (in the 90's), the goal
of the class was to give the student a better idea of how things worked under
the hood to improve their C programming, not to be an assembly programming.

Just today, I helped a coworker debugging a segmentation fault that occurred
in a library where the debugging info was stripped out by looking at the
assembly code.

~~~
inetknght
> Just today, I helped a coworker debugging a segmentation fault that occurred
> in a library where the debugging info was stripped out by looking at the
> assembly code.

Being able to debug applications which don't have debuginfo (particularly
optimized builds) is highly valuable though.

------
paulsutter
The bearishness here surprises me.

The more a company spends on infrastructure, the more they need low-level
people. A good low-level programmer can reduce cost requirements 10x or more.
Any company spending millions or billions on infrastructure can make enormous
savings by hiring the right people. Crucial for Google, Amazon, Facebook, and
even midsize startups can see a big improvement.

And its difficult to fill those jobs. Which means, there's opportunity there.

~~~
chocolatebunny
Can you clarify your statement?What does a low-level person do to improve
infrastructure?

~~~
bitwiseand
Writing and designing code that is mindful of latencies (Amdahl's law).
Temporal / Spatial locality, avoiding needless copies etc.. This approach
needs to start from design and also follow into implementing the code.
Moreover, this is done without compromising on readability.

~~~
CalChris
Umm, Amdahl's law is actually a _negative result_. It's saying that if you
infinitely improve some magical hotspot that is 1% of your workload, the best
you can only ever get to is 99%. If anything, Amdahl is telling us not to
grind on trivialities.

~~~
analognoise
That's not how I understood it - I thought it was about the upper limit of
theoretical parallelism.

But I'm an EE, so maybe I misunderstood some finer points?

~~~
CalChris
An upper limit on the _benefits_. To me, an upper limit is a negative result.
When Leon (in _Blade Runner_ ) finds out that he only has 4 years to live (an
upper bound) he takes it pretty hard.

Amdahl:

    
    
      A fairly obvious conclusion which can be drawn at this
      point is that the effort expended on achieving high
      parallel processing rates is wasted unless it is
      accompanied by achievements in sequential processing
      rates of very nearly the same magnitude.
    

[http://www-inst.eecs.berkeley.edu/~n252/paper/Amdahl.pdf](http://www-
inst.eecs.berkeley.edu/~n252/paper/Amdahl.pdf)

A point made when we read the paper was that Seymour Cray always made sure
that his computers were also the fastest _scalar_ computers even though they
were sold as _vector_ processors.

~~~
analognoise
Thank you for the excellent explanation; I will consider it more thoroughly.

I think mathematicians consider an upper limit a positive bound - positive, in
the sense of being well defined; you're using negative in the other sense? I
actually like that quite a bit.

------
hackermailman
CS:APP 3rd version you can buy on Abe Books the global edition for $20, then
follow this course and watch 'old videos'
[https://www.cs.cmu.edu/~213/schedule.html](https://www.cs.cmu.edu/~213/schedule.html)

Everything in this style guide is covered in CS:APP
[https://matt.sh/howto-c](https://matt.sh/howto-c)

There's also [http://rust-class.org/](http://rust-class.org/) and
[https://www.nostarch.com/Rust](https://www.nostarch.com/Rust)

~~~
bogomipz
Wow, what great links! Thanks. Also I had no idea No Starch had a Rust book.

~~~
kod
The No Starch book is just a print version of

[https://rust-lang.github.io/book/](https://rust-lang.github.io/book/)

~~~
bogomipz
Oh I did not pick up on that, thanks. No Starch are pretty nice nice pressings
though so might be worth it. Cheers.

~~~
steveklabnik
One of two authors here!

Here's the situation: No Starch, being as awesome as they are, are totally
happy with printing an open-source book. The Rust project wanted to do a
second edition of the book for various reasons, so Carol and I have been doing
that, but also, working with No Starch's editors. So they've been
collaborating with us on it. It also means you can get a nicely laid out
eBook, the paper copy, all of that.

The proceeds are going to charity.

------
CliffyA
As someone who can code in C/C++ and was recently looking for work, I don't
think someone should learn low level programming for a career. What I would
consider simple PHP jobs were easily paying more than C++ jobs.

If you look at everyone reinventing the wheel in electron and not caring about
performance I don't think the situation is going to get better anytime soon.

~~~
okatsu
Please don't spread generalizations like these. I'm sorry about the job
situation in your area but I live in a Canadian metropolis where 1) there's no
shortage of low-level work and 2) they will pay much more than your average
PHP consulting shop. In fact, I maxed out very early what someone can hope to
earn at my level of experience.

My point is that your job market is not representative of the whole world and
you're acting as if it were.

~~~
sdflkd
Yet if you did web development you'd (likely) be making 2X+ in America --
provided you were willing to move. Canada doesn't pay.

~~~
okatsu
According to vocal Trump supporters, America doesn't need or want anything to
do with my ethnicity anyways ;)

~~~
sdflkd
Same with me, but Trump supporters are in low volume in areas where you want
to live anyway. :)

------
chimtim
Low level programmer here. This is great but the job market for low level
programmers is very small. Recently, while interviewing, I described a thread
safe, concurrent queue implementation to a hiring manager and he asked me --
"so how does this relate to big data and ML"?

~~~
majke
I struggle to find good low-level devs. It is surprisingly hard to find good C
/ kernel / network programmers.

~~~
itsokimbatman
You're not the only one. My company is desperate for good low level hackers.
We can find people who know Python and Java for days, but people with the
skill set we need are very hard to find.

~~~
dbancajas
i am curious how much is your company willing to pay? is your budget around
150-200K or around 80-120K?

~~~
znpy
asking the real questions

~~~
tropo
Well, sort of. It's forgetting taxes, house prices, and more.

Places where you can buy decent stand-alone homes for $50,000 and/or have no
state taxes are rather different from Mountain View and Manhatten.

It's funny how people in the pricey places look at a $100,000 salary and not
see how it is, all by itself, plenty to support a large family in a home with
a big yard. People living that life look back at the city folk and can't
imagine how they could ever afford the same thing in a city -- it'd be roughly
$2 million (a factor of 40) in San Francisco.

------
pdelbarba
In this thread there's a lot of FUD about there being no job market. One place
to look is to your undergraduate electrical engineering brethren. You'll
basically always be following them around to wherever they go (though often in
slightly higher demand given that board design is typically faster than the
software development phase). Also worth noting that in recent years there has
been an emphasis on FPGA skills for a lot of jobs in the space. It often helps
to have some familiarity, especially if you like working for smaller companies
that wouldn't want to hire a specialist for that role.

------
mtanski
This is great list.

My personal additional to the list would be "What Every Programmer Should Know
About Memory"
[https://people.freebsd.org/~lstewart/articles/cpumemory.pdf](https://people.freebsd.org/~lstewart/articles/cpumemory.pdf)

------
blueatlas
This embedded programming course from UNC Charlotte is quite good for getting
started:

[https://www.youtube.com/watch?v=3Ak8cxN_bLI](https://www.youtube.com/watch?v=3Ak8cxN_bLI)

And, Stan Warford's systems programming class is exceptional:

[http://www.cslab.pepperdine.edu/warford/cosc330/](http://www.cslab.pepperdine.edu/warford/cosc330/)

I believe you can get the video lectures through iTunes U.

For university courses, these are both really good.

------
clavalle
I was having a conversation with a friend working on high volume data problems
and was surprised just how low level he has to get in order to meet some very
stringent but necessary requirements.

I've worked at a high level for so long that I've forgotten how much low level
work I take for granted.

------
LeeHwang
This is great, but does any one have any gentler intros ? I'm trying to teach
low-level programming to one of my junior devs. She graduated from a boot-camp
school and is struggling a bit.

EDIT: Thanks for the replies. Alot of good links.

~~~
Diggie1225
I am taking a great coursera course on the very basics of a computer (uses a
hardware simulator to build the chipsets necessary to build a computer). It is
VERY intro-friendly and gentle, including teaching basic Boolean algebra. Also
can be very hands on, which is what I really like about it. Might be a good
fit!

[https://www.coursera.org/learn/build-a-
computer/](https://www.coursera.org/learn/build-a-computer/)

------
gurugio
Thank you to introduce my page to the world. I've just found this thread and
understood why the traffic suddenly peaked. I'll add several
link/books/courses here to the page. Thank you for all.

------
0xFFC
About career, is low level programming job market that small we see in this
threads comments ? I want to be C/C++/Rust programmer. This is quite
frightening.

~~~
RandomOpinion
Take a look through the monthly HN "Who's hiring?" posts. Jobs of any kind
that aren't web (front-end, back-end, or full stack) or mobile apps are
vanishingly scarce.

It's really quite depressing.

~~~
souprock
Us low-level types don't post there.

We know the bias against defence contractors, against the south, against the
non-urban parts of the USA, and so on. We know we aren't wanted around here.

The jobs can be nice though! My place is looking to hire dozens per year. We
do emulators, JIT, hypervisors, stuff like valgrind, debuggers, manual
disassembly, and vulnerability research. I've been here 11 years. I've worked
with more than 10 different CPU architectures and more than 10 completely
different OSes. I don't normally work overtime, and I get paid more if I do. I
have extreme flex-time. I never have to worry about outsourcing or H1B people.
I'm never expected to take work home or be on call. I get to live in a place
with no state income tax, a stand-your-ground law, almost no crime, almost no
traffic or commute, and houses that commonly go for $100,000 to $400,000.
We're hiring in Florida, Texas, Virginia, Maryland, Georgia, South Carolina,
Alabama... totally the dream for SF and NYC people I'm sure!

Email acahalan, at gmail, if this suits you. Be sure to mention this comment.

~~~
tayo42
How could someone make them selves worth even interviewing for a job like
that? I do Ruby/python and Devops with some small hobby c and Linux stuff in
my free time

~~~
linkregister
Follow the guide in the link. Do the Arduino stuff first; being unable to do
something will drive you to learn exactly as much C and assembly as you need.
Learn to write network code in C. Learn how TCP/IP works and try to do weird
things, such as transmitting information with ICMP (ping) packets.

The next step (for that particular company's work) would be to actually
reverse engineer some complex code and write about it on a blog.

If you want to basically guarantee yourself an interview, reverse engineer and
find a vulnerability in some software, and write about it. Then apply with
that information.

If you're looking for the "easier" way, just apply to DoD in Maryland or
Virginia for an ultra-low-paying, but connected job that allows you to learn
all this stuff on the job.

~~~
tayo42
I was under the impression that a lot of stuff on an arduino was abstracted
away. You mostly set gpio pins to high and low. I have a raspberry pi that can
do that.

~~~
itsokimbatman
An Arduino is just a micro controller with a standard set of pinouts for easy
access to peripherals. You don't have to use the IDE and high level
programming language. At least for the AVR based ones you can write pure C or
assembly, build it with the AVR tool chain, and upload and run it natively.
I'm sure you can with ARM or Intel based Arduino clones too.

------
pklausler
Given two candidates for a job, one that knows assembly language and one that
doesn't, I'll hire the assembly-capable programmer nearly every time.

------
anocendi
I can vouch for Paul E. McKenny's book.

I would recommend anyone with curious mind to read it, even if day to day work
does not involve much low-level programming. It will open up a whole new set
of ideas / knowledge in problem solving toolbox.

------
mhh__
Compiler backends are another topic that can really extends one's knowledge of
their hardware. You end up having to learn a least a bit of every layer
underneath your code e.g. ISAs, Calling Conventions and scheduling.

------
thegayngler
I want to create a new operating system even though we don't need one right
now. So this looks very attractive to me. Thanks for posting.

------
trenzalore
Anyone knows the same kind of roadmap/tutorial but for Windows Low level
programming ? Thank you !

