
Linux debugging tools you'll love - rollulus
http://jvns.ca/blog/2016/09/07/new-zine-linux-debugging-tools-youll-love/
======
partycoder
I think it's a good contribution. Couple of things though:

\- lsof can also print the list of files open by a process. fuser can tell you
which process open a specific file. lsof is mentioned later in the zine
though.

\- sysdig is not mentioned but is great.

\- wireshark/tcpdump/ngrep/etc might be challenging to practically use in the
presence of SSL/TLS. you can mitigate this problem by setting up SSL
termination at your network perimeter so you can monitor unencrypted traffic.

\- additionally to netcat you have curl, httpie and many other clients for
similar purposes. in the Chrome developer tools network tab, you can select a
request sent when visiting a website and click "Copy as cURL". this will
quickly export any request you were making as a cURL command. likewise,
proxies like burp also implement this same feature.

sometimes you need to debug DNS problems, that is not covered sadly. I use dig
for this purposes but I wonder if there's something better.

if you are going to be using command line utilities, getting acquainted with
cat, grep, cut, sed, awk, tail, head, colrm, tr, sort, uniq, comm, wc, etc. is
very recommended. ministat is also cool.

then, about java and node... java has excellent profiling tools, including the
free jvisualvm tool shipped with Java that does the job for the most part.
profiling and debugging node in runtime can be really challenging. specially
analyzing node coredumps with mdb_v8 is not for the faint of heart... you need
to set up a VM with joyent's SmartOS for this. there's an npm package that
simplifies this process called "autopsy". now, flamegraphs might be fine, but
i strongly prefer nodegrind + qcachegrind.

~~~
avtar
_> wireshark/tcpdump/ngrep/etc might be challenging to practically use in the
presence of SSL/TLS. you can mitigate this problem by setting up SSL
termination at your network perimeter so you can monitor unencrypted traffic._

[https://mitmproxy.org/](https://mitmproxy.org/) is pretty awesome for that
type of work.

~~~
mook
In Firefox/Chrome (when using NSS), on developer builds (Aurora/Canary), you
can also set SSLKEYLOGFILE to a path and use Wireshark directly.

[https://wiki.wireshark.org/SSL#Using_the_.28Pre.29-Master-
Se...](https://wiki.wireshark.org/SSL#Using_the_.28Pre.29-Master-Secret)
[https://developer.mozilla.org/en-
US/docs/Mozilla/Projects/NS...](https://developer.mozilla.org/en-
US/docs/Mozilla/Projects/NSS/Key_Log_Format)

------
wyldfire
> ways I've changed how I think about debugging:

> bug is happening for a logical reason. there's no magic

> be confident I can fix it

> talk to someone

Those are good steps. IMO "talk to someone" is a critical step but you'll get
wildly different results based on the quality of your reference resource. Ms
Evans likely has access to someone (or is someone) who clearly knows their
stuff. eBPF and other debugging features enabled by modern kernels are not
well known by many developers.

But "be confident I can [diagnose] it" is a great first step and if you're
persistent you will find the help you need. Sometimes it may take plumbing the
depths of SO or IRC to find that help, but it's out there.

~~~
phil21
Finding someone smart to talk to is important, and a lot of people fail at
this step. I like to consider myself decently good at Linux troubleshooting
and get asked a lot of questions most days.

My sole piece of advice to folks looking for help: Be specific and help
yourself. This means don't come and ask "why is my server slow?!" \- it's
coming into the conversation telling me the exact symptoms and what you've
already tried and where you think the next steps are.

If you are not at that point, you haven't put enough effort in for me to
bother. Those are called paid consulting engagements.

I would say most people get this wrong, and end up being ignored over time.
The few folks who consistently give me interesting (even if they are trivial
and I've seen it before) problems to help them with that they just need a bit
of specific knowledge to solve? I look forward to them contacting me.

~~~
soperj
You don't even necessarily need to be talking to someone smart, or with any
knowledge in the subject at all. Often times I'll figure something out just by
explaining the problem to someone. The act of breaking down where the problems
could be and showing someone what you've tried often leads to more ideas on
how to fix it.

~~~
AceJohnny2
hence
[https://en.wikipedia.org/wiki/Rubber_duck_debugging](https://en.wikipedia.org/wiki/Rubber_duck_debugging)

~~~
soperj
Interesting. Somehow i have no recollection of that term even after reading
that book. It was a number of years ago now... perhaps it's time for a
revisit.

------
AlexB138
The cutesy/manic writing style is a little jarring, but it seems like a lot of
effort was put in to making a useful resource. Worth taking a peak at, don't
let the style scare you off.

~~~
xbryanx
The approachable/playful writing style makes it so much more engaging for me.
It's refreshing to see some of this stuff explained in a conversational and
enthusiastic manner. Obviously, just personal taste.

------
ryancnelson
This 'zine (and really, much of the author's entire website) are pretty much
pure, concentrated joy and tech-geek happiness. I hope my kid grows up to have
the sort of attitude about solving problems and going about life that she
does.

Bravo, Ms. Jvns!

------
jcoffland
While we're on the topic of simple debugging tools. One I find I use all the
time is _top_. Not classically a debugging tool but it sure solves a debugging
problem and I find myself using it very often to check CPU and memory usage.

On the topic of more sophisticated debugging tools I like _valgrind_ and
Google's _gperftools_. These are definitely more difficult to use but I
recommend not giving up on them because the pay off is huge. The trick is a)
knowing how to run these tools and b) knowing how to read and understand the
output. Both can be achieved through practice and RTFM.

~~~
sametmax
I prefer htop to top (or glances if Python is an option), but it's true it's
not installed by default.

~~~
baldfat
Atop is also a great tool

------
wyldfire
Regarding "valgrind" \-- I've had better results with electric fence and
ASan/TSan/UBSan, especially since most of them work on non-x86. And I've had
issues with valgrind when investigating a program that leverages newer x86
instructions than it was built to interpret.

------
albinofrenchy
If you find this interesting, you might also like the embedded.fm podcast she
was on: [http://embedded.fm/episodes/141](http://embedded.fm/episodes/141).

------
oopsies49
Anyone have more details on the netcat file transfer trick on page 9?

You would want to confirm the data received matches the data sent somehow
right?

~~~
1_player
You start a listening nc on the destination server, and push the file from the
source server.

    
    
        Destination: nc -l -p 1234 > foo
        Source: nc destination 1234 < foo
    

Protip #1: You can also send directories:

    
    
        Destination: nc -l -p 1234 | tar xf -
        Source: tar czf - directory | nc destination 1234
    

Protip #2: pv for progress indicators (pv on one side is enough)

    
    
        Destination: nc -l -p 1234 | pv > foo
        Source: pv foo | nc destination 1234
    

Note: netcat (BSD or GNU variants) syntax varies across unixes and distros.
Sometimes it's `nc -l -p 1234`, other times `nc -l 0.0.0.0 1234`. Check your
man.

You can check for data corruption with md5sum or similar checksumming tools.

------
flojo
Sysdig was not mentioned.

Most of all those tools / command, can be achieved with this one tool /
command.

[http://www.sysdig.org/wiki/sysdig-
examples/](http://www.sysdig.org/wiki/sysdig-examples/)

------
helper
This is by far best enjoyed on paper. Print one out and share it with your
office!

------
Davidbrcz
This is awesome !

------
lucb1e
Wow, that is a different style than I'm used to. Must say I'm happily
surprised by good content, despite the playful look. It makes it seem very
accessible, nicely done.

------
lfx
Just wondering if she draw/wrote everything by hand or there is font like
that?

~~~
ramblenode
I am aware of this R package for generating xkcd-style plots and text:
[https://cran.r-project.org/web/packages/xkcd/vignettes/xkcd-...](https://cran.r-project.org/web/packages/xkcd/vignettes/xkcd-
intro.pdf)

~~~
lfx
Actually there is "real" font made from xkcd
[https://news.ycombinator.com/item?id=9302740](https://news.ycombinator.com/item?id=9302740)

------
tbarbugli
I appreciate the effort and enjoyed this paper. One small remark: the first
example is ridiculous. You don't need such a low level tool to find out
someone is doing crazy database queries, just analyze your slow log!

------
garaetjjte
netstat is obsolete.

~~~
hobr
What do you recommend as a replacement? I'd love to get something that wasn't
quite so opaque.

~~~
tangue
Ss has replaced netstat. (By the way each time you're typing netstat you're
suppposed to get a message telling it's obsolete). ss is an improvment on
performance and features, but unfortunately the ui isn't much improved
compared to netstat ...

~~~
khedoros
> By the way each time you're typing netstat you're suppposed to get a message
> telling it's obsolete

Starting about when? Most of the distros I use are a couple years old, and I
don't see "ss" under the package managers.

~~~
t3f
It's from iproute2 - sockstat "ss" Circa ~ 2001, although it's been through a
few maintainers since.

[https://github.com/shemminger/iproute2/commits/master/misc/s...](https://github.com/shemminger/iproute2/commits/master/misc/ss.c)

