
Brushing up on operating systems and C programming - shbhrsaha
http://www.shubhro.com/2018/01/20/brushing-up-os-c/
======
teeray
Can anyone recommend an intermediate to advanced C book? I feel like most C
material I've been exposed to in the past lacks what a modern C project should
have when it comes to best practices, code organization, build tools (should I
use make for this, cmake, where does autotools fit? what about clang vs gcc?),
linters (valgrind?), testing, etc. I recognize that much of this falls outside
of C, but all of these tools seem like things C projects routinely grapple
with.

~~~
earenndil
To answer your questions:

* Clang vs gcc: doesn't really matter. Clang has slightly better compile times. GCC has really advanced but in some cases still has inferior warning/error messages. Best practice is not to really use GNU c features unless you want to. If you do want to, decide if you want to just use the subset supported by clang or the full shabang from gcc. Beyond that: if you want to support both, test on both before release and do release builds using whichever produces faster or smaller (choose whichever metric you prefer) binaries.

* Build systems: plain old make is sufficient in most cases. I would recommend autoconf over cmake just because it's significantly simpler, although cmake has better cross-platform support (it can generate visual studio build files, for instance). I wouldn't use something outside of make, cmake, or autoconf, even if it seems like it's better in every way than those, because it probably isn't, and even if it is, you lose out on the battle-tested, available-everywhere, and widely-used nature of the above build systems. I shouldn't have to install a build system to build your program, and if I do, I should be able to use that build system to build a significant of other programs too.

* Linting: not really necessary IME. Aim for no compiler warnings, though.

* Yes, use valgrind. Anything it complains about, fix. Uninitialized values and out-of-bounds memory accesses are _significantly_ more important and worrisome than memory leaks (because they represent potential attack vectors), but still, it won't complain about something unless it's actually something that should be complained about.

* Testing: like the other commenter said, just a little bit of macro magic and you're golden.

* Code organization: headers in include/, source files in src/. You can separate src/ into subdirectories if your project grows to sufficient complexity that the source files become difficult to wrangle. Separating the headers into separate directories is probably not necessary, however.

* Not directly c-related, but pick a _SANE_ code style and stick to it.

* Debugging: make a separate target that compiles in debug symbols and disables optimizations, and use gdb (or lldb) on it. You don't need much to get started: 99% of the time, all I do is "break main", "run" ("r" for short), "backtrace" ("bt" for short), "frame <number>" (to switch between stack frames), and "print <variable>" ("p <variable>" for short); it's taken me quite far.

~~~
spatulon
I like your thinking regarding using simpler build tools. I took this to the
extreme and now my build tool for personal C/C++ projects is just a file
called build.sh/build.bat. That does little more than:

    
    
        gcc main.c
    

main.c #includes any other .c files that are needed (the term for this appears
to be a 'unity build'). Compiling this way is /really/ fast, which is why it's
okay to use a dumb build script that always recompiles everything.

~~~
keldaris
I do exactly the same thing whenever I write C/C++. Genuinely large projects
aside, I see no compelling reason to waste your time with build systems. This
approach is simple, easy to maintain and lets you get on with writing actual
code.

~~~
ramzyo
Out of curiosity, why not use make for something so simple instead of a Shell
script?

~~~
spatulon
In my mind, shell scripts are simpler. I have no reason to need the extra
complexity that make brings.

Also, I do a lot of programming on Windows, where GNU make would be another
dependency to install. (Also, in my experience, make is slow on Windows, since
they have to emulate fork()). I guess I could use Microsoft nmake, since I
assume it's still installed along with Visual Studio, but again, batch files
are simpler.

~~~
int_19h
Take a look at this: [http://news.dieweltistgarnichtso.net/bin/redo-
sh.html](http://news.dieweltistgarnichtso.net/bin/redo-sh.html)

And for GNU tools on Windows, I would heavily recommend MSYS2 these days -
having Pacman as the package manager is very nice, and there are already a lot
of packages there.

------
git-pull
I recommend (by most expensive, to free):

Marshall Kirk McKusick's FreeBSD Intensive Code Walkthrough:
[https://www.mckusick.com/courses/advdescrip.html](https://www.mckusick.com/courses/advdescrip.html)

Also, _The Design and Implementation of the FreeBSD Operating System (2nd
Edition)_ : [https://www.amazon.com/Design-Implementation-FreeBSD-
Operati...](https://www.amazon.com/Design-Implementation-FreeBSD-Operating-
System/dp/0321968972)

Thirdly: grab a copy of FreeBSD (or OpenBSD) and (a) set it up in VirtualBox
and SSH it into locally (b) use an old ThinkPad. Then grab the source code of
the base system. Build and install it. And start reading code of things like
usr.bin/grep/grep.c

~~~
antoncohen
I have't done the code walkthrough course, but I bought and watched Kirk
McKusick's Kernel Internals course and it is excellent
([https://www.mckusick.com/courses/introdescrip.html](https://www.mckusick.com/courses/introdescrip.html)).
It is based around FreeBSD, but is a generic enough Unix internals course that
it is good for Linux.

I'm thankful to have the opportunity to learn from someone with such deep
knowledge of Unix, who was involved with BSD from the early days in the 80s to
modern FreeBSD.

------
partycoder
This book is not about OS dev but rather systems programming: "The Linux
Programming Interface", [https://nostarch.com/tlpi](https://nostarch.com/tlpi)

For OS development resources, this wiki is very good:
[http://wiki.osdev.org](http://wiki.osdev.org)

~~~
CandidlyFake
> This book is not about OS dev but rather systems programming: "The Linux
> Programming Interface",
> [https://nostarch.com/tlpi](https://nostarch.com/tlpi)

You can also look through the man pages for free.

[https://www.kernel.org/doc/man-pages/](https://www.kernel.org/doc/man-pages/)

Or if you are adventurous you can play around with syscalls in assembly.

[https://syscalls.kernelgrok.com/](https://syscalls.kernelgrok.com/)

Also, the examples for TLPI are available online

[http://man7.org/tlpi/code/online/index.html](http://man7.org/tlpi/code/online/index.html)

~~~
da_chicken
Not to pick on you specifically, but I've really grown to hate this type of
response to this very common question. Most people new to a topic want an
instructional manual or guide, not a technical reference. Man pages and tables
of syscalls are decidedly the latter, and therefore primarily intended for
people who are already familiar with the topics they cover.

~~~
CandidlyFake
> Most people new to a topic want an instructional manual or guide, not a
> technical reference. Man pages and tables of syscalls are decidedly the
> latter, and therefore primarily intended for people who are already familiar
> with the topics they cover

But this thread is about brushing up on OS and C programming. So not novices,
but people who are already familiar with the topic.

~~~
danieldk
Even then... Where are you going to start in reading references? A random
syscall or function a day? I think it is far more useful to e.g. read the late
W. Richard Stevens' Advanced Programming in the Unix environment. It puts
everything in context, provides historical background where necessary, and
gives examples.

Reference pages are not really for brushing up, but more for the 'what was the
address family field of sockaddr called again'-type of questions? Or put
differently: they are external memory.

~~~
CandidlyFake
> Even then... Where are you going to start in reading references? A random
> syscall or function a day?

Sure. Or browse through a bunch of them?

> think it is far more useful to e.g. read the late W. Richard Stevens'
> Advanced Programming in the Unix environment.

That is closer to a reference than a novice tutorial.

> It puts everything in context, provides historical background where
> necessary, and gives examples.

Sure. So do good references. Even man pages do.

> Reference pages are not really for brushing up, but more for the 'what was
> the address family field of sockaddr called again'-type of questions? Or put
> differently: they are external memory.

It depends on your level, experience and your competence in the material I
guess. I'm not saying it's the only thing you need, but in many situations,
it's the only thing you need to brush up.

------
dhuramas
[http://pages.cs.wisc.edu/~remzi/OSTEP/](http://pages.cs.wisc.edu/~remzi/OSTEP/)
is really good and approachable. It was even recommended in one of the Gatech
OS courses.

~~~
monaghanboy
I've gone through the first few chapters of this book, it's fine. The
exercises are somewhat lacking though, in my opinion (the last course I did
was nand2tetris, which was project-based, so perhaps I have unrealistic
expectations). I started learning C concurrently, and didn't find it a
problem.

------
beeforpork
To learn which traps to avoid in C, have a look at SEI CERT C. It's written
very well with examples and explanations, and is structured as a coding guide.

[https://wiki.sei.cmu.edu/confluence/display/c/SEI+CERT+C+Cod...](https://wiki.sei.cmu.edu/confluence/display/c/SEI+CERT+C+Coding+Standard)

------
csnewb
I recently decided to review operating systems, but my old Tanenbaum textbook
from college would take forever to read through especially now that I'm
working full time. I've been watching the Berkeley lecture series on Youtube
(CS 162) and its been pretty good at covering the main topics concisely.

~~~
monaghanboy
What prompted you to review? Does this somehow indicate that you weren't using
the course material in your job(s)?

~~~
csnewb
I graduated from college more than 2 years ago so I forgot almost all of them
CS topics that I've learned and wanted to review the fundamentals (OS,
databases, computer architecture, etc). And no, I don't use of these concepts
very much at work, only surface level knowledge. I've also been trying to
learn more about malware analysis and reverse engineering, and it seems like
having a solid foundation in CS concepts is key to being proficient at it.

~~~
monaghanboy
Would you mind sharing what kind of software you write?

This is interesting to me, since I'm self-taught and seem to have done well in
my career so far (4 years in), but now I'm going through core CS books and
courses, worried that I'll hit a ceiling and wanting to fill the gaps in my
knowledge. I work in devops and infrastructure.

------
lettergram
I personally found the best method of brushing up on C and operating systems
was just walking through examples myself...

Haven't personally done it for a few years, but I went through step by step on
my blog (particularly in 2014):

[https://austingwalters.com/knowledge/](https://austingwalters.com/knowledge/)

I also have a nice repo of a bunch of IPC examples:

[https://github.com/lettergram/IPC-
examples](https://github.com/lettergram/IPC-examples)

------
beeforpork
Knowing and fully understanding the concept of undefined behaviour (UB) and
the difference to unspecified and implementation-defined behaviour is a must
for safe and secure C programming. Because arguably, C and C++ are broken at
the specification level. Chris Lattner puts this in kind words:

[http://blog.llvm.org/2011/05/what-every-c-programmer-
should-...](http://blog.llvm.org/2011/05/what-every-c-programmer-should-
know_14.html)

------
zabana
Somewhat unrelated but I've always wanted ask: What types of small projects
can a beginner start to develop a better understanding of C ? (Beyond basic
syntax, something that would actually make use of memory allocation, pointers
etc)

~~~
AlphaSite
Creating a small UFS implementation (on top of your file system, rather than
as a raw disk is probably easier) in C is a nice fun exercise.

~~~
zabana
Thanks for the recommendation ! Sounds indeed like a fun thing to do.

------
b3h3moth
Great 'free' resources:

\- C Programming Boot Camp
[https://www.gribblelab.org/CBootCamp/index.html](https://www.gribblelab.org/CBootCamp/index.html)

\- aalto-c.mooc
[http://2016-aalto-c.mooc.fi/en/Module_1/index.html](http://2016-aalto-c.mooc.fi/en/Module_1/index.html)

------
django1993
I would also recommend reading Computer Systems: A Programmer's Perspective,
3rd Edition.

~~~
server_bot
Indeed! It's my favorite technical book, several of the chapters have
information I've yet to find laid out in comparable detail anywhere online.

The book was written for a CS course (15-213) at Carnegie Mellon University,
course materials here (supplement, but no replacement for the book):
[https://www.cs.cmu.edu/~213/](https://www.cs.cmu.edu/~213/)

------
stiff
I like this little book, in very small amount of space it teaches you how to
build a bare bones OS, from there you can start doing your own experiments
which I think is the best way to really learn something:

[https://littleosbook.github.io/](https://littleosbook.github.io/)

It's good to not only know operating systems in general, but also to
understand some of the internals of the operating system you are actually
using. For Linux, this is good:

[https://0xax.gitbooks.io/linux-
insides/content/index.html](https://0xax.gitbooks.io/linux-
insides/content/index.html)

------
andlrc
TLPI is a fantastic reference book to have when written code.

------
orsenthil
CS50 courses (This is CS50!), cs50.io is currently one of the easiest and the
best introduction to C language. It's very modern approach to teaching C
programming.

------
swinghu
mark as it is ,learned a lot

------
sifoo
Will recommending my own project get me hell-voted? Cixl is a straight forward
3-stage (parse/compile/eval) interpreter that is designed to be as fast as
possible without compromising on simplicity, transparency and flexibility. The
codebase has no external dependencies and is currently hovering around 7 kloc
including tests and standard library. There are some articles explaining
design decisions, but I've been mostly busy with code. The code should be
perfectly readable though, and contains plenty of good ideas distilled from 32
years of daily practice.

[https://github.com/basic-gongfu/cixl](https://github.com/basic-gongfu/cixl)

[https://github.com/basic-
gongfu/cixl/tree/master/devlog](https://github.com/basic-
gongfu/cixl/tree/master/devlog)

~~~
Bjartr
Could make a good show hn post

~~~
sifoo
Been there, done that :)

[https://news.ycombinator.com/item?id=16081592](https://news.ycombinator.com/item?id=16081592)

~~~
akkartik
Yes I followed that thread with interest, as well as the earlier one on
Reddit. I still have trouble following many of your statements, though.

Here, are you saying the Cixl _implementation_ is a useful example of C
programming? Or something else?

