
Rabbit Holes: The Secret to Technical Expertise - dailymorn
http://blog.bityard.net/articles/2019/August/rabbit-holes-the-secret-to-technical-expertise.html
======
cle
The trick is knowing _how deep_ to go, which depends a lot on your goals. If
you just want to learn for your own curiosity, rabbit hole forever! If you
want to build useful, applicable knowledge, there’a a peak of usefulness after
which there are diminishing returns. So being self-aware enough to know when
to pull the plug on your rabbit hole is pretty important (professionally).

Also certain kinds of rabbit holes are way more useful than others. If you’re
rabbit holing into some obscure, arcane system, and the stuff you’re learning
isn’t challenging you _fundamentally_ , you’re probably wasting your time (I
call these “rat holes”). If your rabbit hole will yield new fundamental
insights into something important...now that’s a good rabbit hole!

~~~
Robotbeat
The best depth before you hit the knee in the curve is undergraduate-level
textbooks. Sometimes you have to go deeper, but that's the level that is
presented in a form that's easy to grasp and can be quickly applied without
using too arcane language and abstractions.

Also: Handbooks. For instance, the Art of Electronics. And I recently picked
up a mechanical engineering handbook, and the breadth of the topics covered at
a level I can quickly understand is very refreshing. (I'm a physicist, but I
mostly do mechanical engineering and technician work, with a bit of aerospace
and electrical engineering and material science.)

And combined with youtube and Wikipedia, you can learn a lot very quickly.

~~~
voltagex_
>Art of Electronics

What would you suggest before The Art of Electronics?

~~~
zrobotics
Practical Electronics for Inventors
[https://www.goodreads.com/book/show/687619.Practical_Electro...](https://www.goodreads.com/book/show/687619.Practical_Electronics_for_Inventors)

Title is slightly wanky, but it's the best non-textbook beginner guide I've
found, since it actually does contain all the math & theory a hobbyist will
need. The explanations are very clear and most motivated people with high-
school math should be able to follow along. It's not the best to read back-to-
back, but has detailed explanations of lots of areas of electronics that make
great primers/refreshers. I had it out last week for a quick refresher on
notch filters. Doesn't go into tremendous depth, but more beginner/hobbyist
friendly than Art of Electronics shich is similar (and excellent) but aimed at
EEs instead of hobbyists.

For something even more beginner-oriented any of Forrest Mims' books (formerly
published by Radioshack/Tandy) are great but a little dated, and don't cover
anything more modern than discrete logic and don't provide much in the way of
theory. Excellent introduction to electronics books though, many fun projects
for getting kids into the hobby who want to dive a little deeper into
electronics vs. hooking up modules to an Arduino.

------
HenryBemis
Shortcuts is my personal mortal enemy. I talk to people that "really want to
learn topic X" but: a) don't want to read any books, b) don't want to watch
any training videos (from nuggets in the old days, to youtube tutorials now).
They just want a quick and dirty solution to everything, always. That comes
with a price.

People want shortcuts. I don't like or use shortcuts 99% of the time.

My experience is that when you go looking for X to serve a need today, you
discover Y and Z and guess what.. Y and Z will come very handy tomorrow, when
in a call/meeting about ABC, miraculously Y and Z are similar and you speak up
a d impress everyone and get the job/contract.

~~~
stevekemp
I frequently get emails from people which follow a predictable pattern:

"I'm doing a PhD in .. I saw your blog post .. can you point me at a tutorial
for advanced-subject?"

I often wonder why people don't do research, and what would happen if
tutorials just disappeared. People would have to do their own research and
learning. Maybe things would take longer, but I think the people would benefit
from it.

~~~
jplayer01
Admittedly, I don't send personal emails, but on HN or Reddit, I'll often ask
experts or people who are at least knee-deep in a topic themselves (or find
previous instances where expert(s) was asked).

It's not about laziness, it's just that as somebody who doesn't know anything
about established papers or books of a certain field or how it's structured,
it can help a great deal to get started if somebody can point in a general
direction of where to spend my attention (which is a limited resource). As
somebody with dozens of interests across a bunch of disciplines, drawing on
existing knowledge has been such an immense time saver and help that I'm
always grateful for. There are a _lot_ of books and papers out there. And not
all of them are particularly good. If other people have experienced them and
share that experience, why shouldn't I want to draw on that?

Also, I've gotten a response as often as I haven't. And when I don't, it
really isn't a big deal. You don't need to respond to every request.

~~~
stevekemp
> You don't need to respond to every request.

True, but I feel a little bad if I don't at least say "Please write a tutorial
yourself, people would be pleased".

Sometimes useful discussion occurs, but mostly it feels like "please send me
the codes / do my homework for me". It's hard anyway to judge intent.

~~~
TheOtherHobbes
I'd love to know what percentage of the population has any interest in self-
directed learning.

It may well be smaller than we hope. And HN - probably software in general -
is unlikely to be representative of the population median.

------
acangiano
Reminds me of how, as a kid, I tried to solve Fermat’s Last Theorem. I didn’t
have a chance in hell, of course, but my attempts enabled me to learn a ton of
advanced math and even finding on my own some results that turned out to be
already well known (e.g., Euler’s formula). It's one of the purest forms of
learning but as an adult, I no longer have the time or freedom to do it to the
same extent. Still, the basic approach is worth keeping. You just need to
limit yourself a little so that you don't end up too deep in the rabbit hole,
spending a ton of time on things that you really don't need to know.

~~~
codingslave
I had a similar approach with the following equation:

What does the infinite series of 1/(k^2) converge to for k=1 to infinity.

When I was in high school, I spent weeks going down this rabbit hole of
mathematics, failing to solve it. Years later I discovered that this equation
is a special case of the Riemann Zeta Function, which was also first studied
by Euler. His proof is really elegant. The funny thing is that with some math
equations, sometimes you intuitively can see that there is something important
or far reaching about an equation, even if you arent skilled enough in
mathematics to discover it. Some kind of curiosity draws you in

------
LogicX
Or positioned another way, this is Yak Shaving:

Any seemingly pointless activity which is actually necessary to solve a
problem which solves a problem which, several levels of recursion later,
solves the real problem you're working on.

[http://yak-shaving.urbanup.com/1164318](http://yak-
shaving.urbanup.com/1164318)

------
Tepix
The "right" solution to get a quiet login is to

    
    
       touch ~/.hushlogin
    

in your home directory.

It's documented in the login(1) man page.

There's also a global /etc/hushlogins file you can create to take care of all
users.

------
tombert
Rabbit holes are why I grew to love Haskell and pure-functional programming in
general. For better or worse, Haskell (Agda, Idris...) doesn't let you
compromise very often.

You could argue that this is terrible for productivity, and you might be
right, but at the same time, means that a lot of people have done a lot of
research on how to do things the "right" way in Haskell, and any project I've
done with the language has led me to learning a whole slew of new things about
type theory, category theory, and PL theory in general, which I can then take
and apply to "productive" languages.

~~~
shanemhansen
This aligns precisely with a memorable interview I gave to a PhD student who
wanted to do an internship at a startup.

"The thing I love about Haskell is that even the simplest things become
research projects"

However I understand where you are coming from. There's something to be said
for environments that encourage learning.

~~~
tombert
Every now and then I get mildly embarrassed because someone has managed to
summarize my long-windedness to a beautiful and elegant sentence. The line
"The thing I love about Haskell is that even the simplest things become
research projects" has done that. I'm going to steal that.

------
esotericn
This article describes pretty much the entire reason I got in to software
development, why I'm still doing it, why I'm here today, why I run Linux on my
main machine, etc.

I have an undergraduate degree in Physics for largely the same reason.

I'm increasingly convinced that this is just a mindset you have or you don't.

For me, the world is a place to tinker about in, learn, build stuff, break
stuff, etc.

I'm currently tinkering about with my homelab setup, tweaking the security,
setting up redundancy, etc. Why? 'cos I can, and when you hire me you get all
of that.

It's certainly served me well so far!

------
mustntmumble
As one who simply tinkers with Linux but is not a guru, I found this article
quite inspiring! I had no idea just how deep that sort of rabbit hole can go.
I feel inspired to try to delve behind the scenes for myself... The author has
made my day :-)

------
jaequery
Yeah this is just overkill. I haven’t edited /etc/motd in a while so this was
news to me but this is just code creep and makes me sad.

~~~
hedora
Same here. My reaction at each step is “how do you delete this subsystem?” and
“what poor person wrote this?”

Still, nice learning exercise for the author.

~~~
pjc50
Exactly. I'm sure somebody wanted it at some point somewhere, but now it's on
every system. No wonder people keep starting new minimal Linux distros.

------
lordnacho
This is the reason why people who don't respect coding as a discipline end up
producing terrible code.

In my professional life I've run into a lot of people who will say "the idea
is already there, it just needs to be coded up".

They will then come across common coding issues like keeping things in version
control, adding tests, network config, and language specific things. They
decide they don't have time for this particular rabbit hole, because of course
they already know what they want.

The good people will over time end up investigating everything, even if it
doesn't seem relevant to your LOB application. And they produce much better
code because of it.

~~~
jjeaff
As someone coding for my own business, I often find myself deep down a rabbit
hole and have to get out and force myself to just use that ugly patch.

It's a balancing act, because it hurts me to just leave something without
completing it "the right way" but I have found that if I try to do everything
"the right way", I would get so little done.

I try to keep this in mind when reviewing or working on other's code. Some
things will anger me at first glance but the problem is, they were probably
just prioritizing different things at the time than I would have prioritized
right now.

------
sharadov
I love this kind of stuff, it's the most pure kind of learning, no end goal
but satisfying your innate OCD. I've gone down these rabbit holes in tech and
alternative fields - politics, philosophy, cooking. It's crazy, but it lead me
to some life long hobbies.

------
pjc50
I'm not sure about "technical expertise", as most of the stuff you learn while
doing this is highly specific, but perhaps the real expertise is having the
toolset and determination to do this at all. To keep taking screws out of the
system and removing the covers until you find out where the system makes a
particular decision. Fortunately in this case you can go to the source; if you
can't, or as a first attempt, strace will usually tell you where a particular
program gets its information from. Harder to use here as you'd need to setup
the login process itself with strace and there's privilege boundary
considerations.

One observation I would make is how many layers of "plugin architecture" have
been added here. Probably the original SysV login program just printed
/etc/motd. Now we have PAM, and pam_motd itself also runs a bunch of tiny
plugins, some of which also refer across to systemd.

I should probably write up some of my own deep-dive debugging stories, if I
can still remember enough detail. The "why does structured exception handling
sometimes not work on Windows CE" was a particularly time consuming one which
involved disassembling bits of Windows kernel.

------
simosx
There was drama about the MOTD two years ago.

Here is the summary from LWN,
[https://lwn.net/Articles/726902/](https://lwn.net/Articles/726902/)

------
dusted
So, blur the hostname in the SSH command, login to machine, where the hostname
is ip-172.31.48.7.. Two things are funny here: First, the failed attempt, and
second, the idea that anyone cares what the IP of your raspberry pi is.. If
anyone cares to know, mine has 192.168.30.12.

~~~
reportgunner
haha, perhaps there was some profanity that had to be blurred out

------
cr0sh
Rabbit holes and their implications aside...

As a software engineer, a part of me has died because of the hardcoded strings
(in the pam_motd.c patch). Why aren't these defined as constants in the header
file?

Hopefully there's a good reason...

------
mkl
The metaphor critique paragraph seems to be missing the fact that name comes
from _Alice in Wonderland_ , where the protagonist falls "through a rabbit
hole into a fantasy world":
[https://en.wikipedia.org/wiki/Alice's_Adventures_in_Wonderla...](https://en.wikipedia.org/wiki/Alice's_Adventures_in_Wonderland)

I.e. it makes complete sense in context.

------
cr0sh
Most of us have fell down these rabbit holes of learning before; the author
does a great job of illustrating the concept, showing the solving of an
"ultimate question" about how/why something works the way it does.

I don't know if it counts the same, but I once went on a weird journey to
solve a particular coding problem that I was certain was already answered. I
don't recall the details, but I do recall thinking to myself "This seems like
a common problem; a quick google should bring up a solution..." \- so, one
google search later, and:

[https://xkcd.com/979/](https://xkcd.com/979/)

At that point, I vowed to solve the problem - again - and post the solution
(of course the fact that I can't recall what the problem was negates me being
able to show you what I mean - my apologies - sigh).

I quickly found that to solve the problem, I first had to learn how to code X,
which in turn meant learning about Y and Z, and also coding A, B and C. I
ended up going down these rabbit holes; by turns learning and coding partial
solutions, by others finding example code I could adapt to my needs (not
always in the language I was working with, either). I spent more than a few
hours on this task, but I eventually did come up with a solution.

And as I recall, I did post that solution to the thread where I encountered
XKCD-979 - so future people wouldn't be inconvenienced.

------
drawkbox
We use platforms and layers that are at the level of branches and leaves on
trees, supported by abstracted away structures hiding complexity that make
them simpler to use as a platform higher up the tree (or down).

A researching traveler can go down the branch, to the trunk with lots of time
rings and layers, and then further down there are many root systems and rabbit
holes.

All sorts of functionality is abstracted away in layers down there, all the
way to machine code and binary and everything before the epoch 1/1/1970\. Epic
programming and technology pioneer legends created it, they took complexity
and made a simple platform layer for others to build on.

------
squirrelicus
> Unless you got into tech solely for the money (you monster), it's stuff like
> this that we nerds live for.

I take offense to that! :P

The rabbit holes of my past are half of why my paycheck is where it is. (The
other half is interpersonal skills)

------
peterwwillis
It seems like the difference between a complicated, difficult system and a
simple, easy system is whether it changes over time.

I feel like a Linux kernel in a NetBSD userland might be the best sweet spot,
OS-wise, but I haven't used NBSD in a long time.

------
jimnotgym
> * Keen to learn Istio? It's included in the single-package MicroK8s.

But this is a rabbit hole! His rabbit hole is about removing a rabbit hole.

------
prometheus76
I'm going to go meta with my comment and say that this article is a fantastic
object lesson as to why linux faces extremely low odds of ever being adapted
as a widespread replacement for windows or OS X. How many people (outside of
your coworkers) do you know who would or could follow a path similar to the
one laid out in this article?

I'm speaking as someone who has dabbled with linux since 1998. At that time,
and now, there is a huge, confused ball of conflict between linux
distributions, various levels of customization to those distributions, and
documentation available on the web.

Here's a quick, recent example from my own experience. I decided I wanted to
learn more about Emacs. I work in Windows at work, so I downloaded portacle.
Great. Works great. Then you start watching youtube videos on how to customize
Emacs, but most youtube videos are using Emacs in Linux. So you then have to
find where portacle puts all the stuff that is in a completely different place
than all the Emacs documentation and videos say it is. Great. And I haven't
found anyone who talks about the benefits/costs and gotchas about whether to
edit the init.el file, or the .emacs file, but now we're getting into the
weeds.

So after that frustrating rabbit hole, I think, "I'll just install a linux
distro in a VM." Next thought: "Which distro should I get?" Search engines are
no help when it comes to finding the "best" linux distro because all I can
find is superficial opinion pieces on the matter.

So back when I worked at an ISP in the early 00s, they used Debian, so I
download Debian.

Try to install emacs in Debian, and I start getting weird errors about some
certain package not being current enough, and apparently someone isn't
maintaining/updating the install script for Debian. That's three days lost
getting to that conclusion.

So I wipe that VM and download Ubuntu, because it seems to be the most widely
used distro, and seems like a good distro for "normies" like me. Install
emacs, no problem. Can customize by following youtube videos, no problem.
Enjoying emacs now.

All of this comes at a high price, however. I've been messing around with
trying to use emacs for a few months now, on my side time at work and at home,
and it's only been in the last couple of weeks that I've been able to use the
package in a way in line with the available training materials.

Normies have no kind of patience for this kind of ridiculousness. Techies
justify these "rabbit holes" as enthralling and fun, and yeah, it feels good
to finally figure some of this kind of stuff out, but overall, it's a massive
distraction from actually USING the tools, rather than just futzing around,
trying to get the damn tools to work in the first place.

~~~
drKarl
If your're on Windows 10 you could've used LSW (Linux Subsystem for Windows),
on older Windows something like Cygwin. When I have to use Windows for work,
Cygwin is amongs the first things I install (actually I found a Cygwin
derivative called babun pretty good, unfortunately I think it's now no longer
maintained). I use Vim rather than emacs, but emacs works well too.

