Hacker News new | past | comments | ask | show | jobs | submit login
Writing System Software [video] (youtube.com)
505 points by dsr12 on Mar 15, 2018 | hide | past | web | favorite | 72 comments

In case you are interested in providing feedbacks about the next episodes, my current list of potential ideas is:

1. Code snippets: show small self contained functions from random places inside Redis.

2. Redis Sentinel: the first design, the immediate redesign of v2, and how it is internally implemented.

3. The use of goto in Redis.

4. Redis protocol parsing and zero-copying of objects.

5. The odd Redis persistence model: fork() and fighting the copy-on-write.

6. Redis crash report. Generation and usefulness over the years.

7. Observability inside Redis: latency hooks, LATENCY DOCTOR and MEMORY DOCTOR.

8. Hybrid data structures: Streams macro nodes implementation.

9. RIO: Redis I/O abstraction and OOP in C.

10. Redis Cluster: how the Redis message bus is implemented.

11. Redis Cluster: hash slots tracking and redirections.

If you have any preference and/or other ideas, will be cool to hear.

About this experience itself, good things: shooting a video takes more or less just the time to record. Preparation is just a few minutes, and later post processing is an automatic process... So in 1 hour max everything is done. However I see that the reach of the videos is very low compared to a blog post, which requires more time to write sometimes, at least to have the same amount of info. So it's a tradeoff. Also consuming the video may be a problem for many, for instance for language barriers due to the speaker accent (in my case) and the listener ability to understand English without subtitles. Let's say I want to make 10 episodes and re-evaluate if it's a good idea to change style and do 5 minutes episodes instead.

> language barriers due to the speaker accent (in my case) and the listener ability to understand English without subtitles.

It's usually easier for a non-native English speaker (at least below ILR[1] Level 3 or CEFR[2] Level B) to understand English spoken by another non-native, because the pace is slower, idioms and expressions are not used so regularly, etc. Your accent is much easier for non-natives than a strong Irish accent, for example. So I wouldn't worry about that too much.

[1] https://en.wikipedia.org/wiki/ILR_scale [2] https://en.wikipedia.org/wiki/Common_European_Framework_of_R...

Fully agreed. I tend to find non-native English speakers' accents more understandable than many native British or American accents, ironic as it is. Salvatore's accent is quite easy on the ears.

Same goes for a tendency of many native speakers to indulge in idioms, archaic vocabulary and convoluted syntax just for the heck of it. Non-natives just cut to the chase, which is exactly what you want in a technical discussion.

Cool, makes sense, thank you. I just want to talk slower in the next videos, so that it will be easier... at the cost to enlarge the duration a little bit.

Playback speed is adjustable so I wouldn't stress too much about it.

Don't worry about the language. You are very easy to understand and follow. Great work!!! Also I enjoyed seeing you in video. Best regards A random fan :-)

What's interesting to me about the videos vs. the blog posts (I like both) is that only the video shows me the way you work. I'd love to see a video where you recreate finding a bug, how you worked to go from issue to repro to fix, just to see more of how you work.

Yes, this is definitely the power of the video, to explain the same in a blog post would be like a chapter, showing in this case is simpler. I'll consider doing bug hunting recording the whole session, and then editing accordingly (because it takes hours or days... for the interesting bugs).

Yeah, that is the hard bit of it for sure, that's why I thought maybe you could just sum up and demonstrate the steps you took after the bug hunt was over. But an edited version would be very cool too.

As an HFT systems programmer myself, 5, 10, 11 are interesting to me. As in, using Unix primitives to persist state in a distributed system.

It seems like there are really 2 meta topics: 1: Using C and the OS to maximize performance 2: building distributed systems

One vote for #5. Also a general suggestion: these sessions could be livestreamed on twitch.tv for live interaction with viewers. Of course the VOD can still be uploaded to youtube later.

How about writing a book with all these potential ideas and name it "The Design and Implementation of Redis in-memory Database"?

I wish I had the time... In N years hopefully I'll switch into writing much more, now I've a book about C that will never end while I work constantly at Redis for instance. But for now me doing first-hand coding is what the project needs... eventually I may probably switch more to a design figure, but will take time. However soon or later I may take a few sabbaticals, so that I can continue working at Redis as I do now but pausing, for instance, for 1 month every 4 months or alike, doing this kind of projects.

I've always loved your LOAD81 project and would encourage you to return to it and add some new features - so maybe how you made it, what it can be used for, how you can extend it by adding functionality .. ? EDIT: I know its not as sexy as your Redis work, but I think there are a lot of wise nuggets in this project that other developers could learn from.


Thank you, I may indeed use some non-Redis part of code for the next episodes.

> However I see that the reach of the videos is very low compared to a blog post

That's the reason why in my other comment, the data screencast I created was made in addition to a blog post, where the blog post and the video explicitly link to each other as cross-promotion.

That does extend production time significantly though.

So cool! Redis is one of my favorite pieces of software and it's super interesting to see some of the internals explained more clearly. At the moment I'm only on the second episode, but I look forward to the rest of them. Perhaps replication could be a topic as well, if it's not already covered in the Sentinel and/or Cluster episodes. Another thing that would be interesting is how connections are handled and how concurrent commands from separate clients get executed in a sane way.

As an aside, I noticed several typos sneaked into comments here and there ('command gets replicated to the salves', etc). I'm half considering filing a pull request for them, but feel it might come across as overly pedantic. Would such a PR be appreciated?

Great ideas! Thanks. Sure the PR fixing the typos will be accepted (I'm very slow with PRs usually, but the typos are faster to merge if you follow my hints later). Better if they are small PRs with a few typos fixed because many files are continuously changing otherwise a big huge PR will end conflicting in many places.

One vote for #4, not sure how related it is to Redis but I always have fun showing how much snappier flatbuffers is vs protobuf because they don't need to allocate/copy/etc when reading data off the wire.

I feel like that knowledge needs to be shared around a bit more.

Thanks for the videos, it's always a pleasure watching good engineers work. A lot to learn.

As a side note, can you share your Terminal color scheme? It's really nice

This is inspired to the commodore C64, I just picked a few random colors with the picker in the terminal. Unfortunately I lost the color scheme the previous week while in Tel Aviv, since this color scheme does not work well when projected so I had to switch to B/W and the simplest way was to just modify the current color scheme.

Do you do all your coding on a laptop with no external monitor?


5. The odd Redis persistence model: fork() and fighting the copy-on-write.

Only thing I have to suggest is changing your sound recording setup to reduce the echo.

Interesting stuff, I haven't had to write C++ in a few years, it's nice to have a bit of a refresher.

Just hanging fabric on the walls should make a world of difference.

I'm enjoying this very much too, but the bell is jarring.

I'm actually waiting for S2E01 :)

I would be very interested in 2. I know some other folks who be as well. Cheers.

Please add 1. Code Snippets.

Please show me how Redis rebalances data when more shards are added.

Having watched similar series in the past, I would highly recommend some extra effort be put into the presentation - recording videos with no attention to lighting or sound makes it really hard to see and hear what's going on. Specifically for me, it's really hard to understand what's being said over the keyboard and reverb.

Some basic lighting (so your face isn't in shadows), a separate directional microphone (even a $40 snowball would help immensely), and potentially some sound baffling in the room would make a real difference to the video quality. Finally, dark colors with low contrasts encode quite poorly on YouTube; some of the code is rather hard to read.

I realize we're watching a software engineer and not an entertainer, but the payoff in the quality of the videos for relatively little effort and investment is remarkably big.

I second fixing the audio. If $50-100 USD for a nice setup is too much, then for $7 off amazon you can get a clip-on, lavalier microphone for your shirt that sounds a lot better and plugs right into your computer.


Or just pair a bluetooth headset and use that instead of the mic that is physically attached to the keyboard.

As someone who has spent a fair amount of time playing with this, post processing audio is a big win. I use audacity and perform the following effects (the order is important):

- Noise reduction

- Compressor with "Make-up gain for 0 db after compressing turned off".

- Equalisation with a curve that starts at -6 db at 20Hz, peaks at 6 db at 150 Hz and then curves down to -30 db at 20000 Hz.

- Normalise with Remove DC offset and normalising at -0.1 db (0.0 db sometimes leads to problems for some reason).

- Amplify if you need it (usually I don't)

The logic here is: Noise reduction, to reduce hiss, key clicks, etc, etc., Compressor to squash the low and high amplitudes (turning off making up the gain also forces it to squash in a certain direction... I forget which at the moment, but it's important), Equalise as a kind of nicer low pass filter that gets rid of the noise reduction artefacts and other noises that make it through, Normalise to bring the amplitude back up that was knocked off by the compress (but in a nicer way than amplify would give you). Finally just fine tuning with amplify.

I usually re-encode in FLAC and then mix it back into my video using my video editing software (I use Kdenlive).

The difference in audio quality is unbelievable. You need to fiddle around a bit with the settings for your voice/environment, but I highly recommend giving it a try.

> - Equalisation with a curve that starts at -6 db at 20Hz, peaks at 6 db at 150 Hz and then curves down to -30 db at 20000 Hz.

this seems like a very bad idea, this will give you very dull, boomy sound, for most recording setups. It's a bit too much work to type out everything here, but this is not good advice.

Most of the time the best improvements are to be made with getting a decent microphone and set it up properly (doesn't have to cost much more than $100).

I haven't had any issues with the audio. How are you listening to it?

The voice is reasonably clear but the loud keyboard tapping is a problem. A voice microphone not attached to the keyboard would help that.

I really feel bad about blatant self-promotion; especially in a thread like this about somebody else who is producing video content. However, I think some folks here may enjoy my work.

I have a daily programming stream on Twitch: https://twitch.tv/nybblesio. I stream from 5am-10am MT Monday to Saturday. (I have to take at least one day off or my brain overheats.)

I also archive all of my streams to my YouTube channel: https://www.youtube.com/channel/UCaV77OIv89qfsnncY5J2zvg

I have several projects running concurrently and I switch between them each week. This week, I'm working in C++17 for Ryu: The Arcade Construction Kit. Next week, I'll be writing ARM AArch64 assembly language on Raspberry Pi 3. I also have an MS-DOS arcade game project written in 100% x86 assembly language. I worked on that last week. All of these projects are curated into playlists on YouTube if you want to check them out.

i confess i am completely not understanding this trend in twitch streaming coding. i prefer asynchronous videos where i can speed or fastforward through parts. i understand the value in seeing people make and solve mistakes but that can’t be the only value in -watching- a live stream can it?

(is this what it feels like to be old?)

I think the essence of any live streaming is an active chat; viewers ask questions, point out mistakes, &c. you 'engage' w/ your audience irt.

There's nothing wrong with self-promoting on HN. In fact it's not far from the sole point of HN.

Though it's amusing when people come out of the woodwork and call you a spammer. That happened to me once. It's like... Yeah, but that's what we're all doing here. What's the point otherwise? We all build stuff and we all talk about it.

That sounds great! Thanks for sharing.

It's really fun to see that the people that we consider geniuses are really just smart people spending their lives forming concepts into reality, one function at a time, like the rest of us.

OTOH, smart people often build systems with excessive complexity, leading to maintenance problems. Genius is found in making complex things comprehensible and simple.

I would say that people who build systems with excessive complexity are (almost per definition) not smart. They made appear confident and have a high level of "I am so smart" feel about them. But they are clearly not.

Some people learn the value of simplicity through experience. They start out building over-complex things, and they’re not “not smart,” just inexperienced and misguided.

Excited to check out this series and glad that Antirez took the time to make it. Would be cool to see more people make this kind of thing in the same mold of this and the DHH series from the last few weeks. Very interesting to hear something like a person's inner monologue while they're writing and reading code.

I've been trying similar-style videos on advanced data analysis/visualization since such screencasts are an unexplored area (my most recent video using Stack Overflow data: https://www.youtube.com/watch?v=6OeyTfp7eq8) but there are a few snags which make it less accessible for anyone to do:

1. It requires subject preparation to avoid going off on a tangent. (in the case of the video above I cheated and use prewritten code, and the video was still 30+min long.)

2. It requires specialized software and hardware that's not free. (good recording software + good mic + good webcam are $50-100 each)

3. It requires good hardware preparation too (e.g. precise placement of the mic and good light for webcam, so I couldn't record at night)

4. It requires time for editing (unless in my case you slap on background audio and call it a day)

I bought a new mic since I was unhappy with the audio quality in that video, and am taking steps to refine video focus.

>good light for webcam

Studio lighting would solve that and falls into your mentioned price range per item.

What recording software do you use?

Screenflow (https://www.telestream.net/screenflow/overview.htm), which is the same software used for OP's video.

I missed the DHH series. Do you have a link? I haven't been able to find them.

The series moved to a channel called "Getting Real" where Jason Fried also publishes videos about design decisions.


Yes, I wonder if Antirez is inspired by DHH making these videos.

He says exactly that in the first 15 seconds of the video.

Indeed I totally followed DHH example, it looked like a great thing if done by multiple persons and personally I would like to see videos like that in different areas of design than programming.

"This is about system software, so we will see lower level things [...] to show --in real source-code, not in small toy examples-- how things work, why certain things were implemented in a given way. You know, when software starts to become bigger, then the very clean approaches start to fail. You have to face compromises in order to make things work.."

Thanks everybody for the feedbacks! I'm going to improve my mic setup :-)

Excellent series. English is easy to understand and the thought process is simple to follow. The only suggestion that I have is to make the videos a bit shorter. 10 to 15 minutes will work great for me.

Very good work. How about a video/blog where you can describe use-cases where redis has been creatively used to solve a tough problem. With some redis design patterns included?

That would be great but it would be more a work of collecting data from other people building things with Redis, and I've very very few interactions with the many people running Redis. I kinda do my part in the Github repository and very seldom I've interactions with other folks. I'm happy with this setup because interactions = not working.

Nice work @antirez, you're an inspiration to the community!

Thanks! I started the video series thinking nobody at all would care, so it's nice to see that it's worth to continue.

Thanks a lot Antirez for doing this! I have been a fan ever since you discovered the idle ping scan! Your work has inspired me to keep growing!

I thought I was the only borland's blue background widow out there.

It immediately reminded me of my days of dabbling in QBasic.

For people wondering if Antriez is following DHH's example, he is. He announced it in this tweet: https://twitter.com/antirez/status/966052769391153155

DHH wrote about his series in this blog post: https://m.signalvnoise.com/on-writing-software-well-aee37807...

Link to his playlist: https://www.youtube.com/watch?v=H5i1gdwe1Ls&list=PL3m89j0mV0...

That's no longer the correct playlist. Watch the last video of that playlist.

The correct playlist is now this: https://www.youtube.com/playlist?list=PL9wALaIpe0Py6E_oHCgTr...

Having a hard time getting past the blatant use of vi instead of emacs... ;-)

I'm not strongly in one side or the other btw ;-) Just happened to start with vim feeling very well with it, and continued for years. From time to time tried Emacs back in the days (20 years ago, like), and was nice but could not found a reason to switch, very different paradigm.

Yeah, I'm just teasing a bit... which is apparently not much appreciated.

Registration is open for Startup School 2019. Classes start July 22nd.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact