
How does gdb work? - deafcalculus
http://jvns.ca/blog/2016/08/10/how-does-gdb-work/
======
DannyBee
"Anyway my impression is that DWARF is a large and complicated standard (and
possibly the libraries people use to generate DWARF are subtly incompatible?),
but it's what we have, so that's what we work with!"

Having written at least 4 complete DWARF readers and writers (GCC's location
list support, GDB's expression evaluator, the thing that became google
breakpad's debuginfo reader etc), it's really not that bad.

In fact, compared to pretty much any other debug format, it's wonderful. All
of the forms are consistent, and outside of the index tables, and a few places
where backwards compatibility was needed (ie the world moved from 32 bit to 64
bit, but before that, DWARF supported 64 bit _sized_ debug info on 32 bit
processors), the encoding is sane.

libdwarf, on the other hand, is ... not so much. I love david a, and (AFAIK)
he's been working on DWARF since SGI was at 1600 amphitheatre parkway, and
keeping libdwarf up to date. It's one of those open source projects nobody
ever realizes has been around 20 years and that someone has kept it working
great (see ftp://ftp.sgi.com/sgi/dev/davea/objectinfo.html)

However, libdwarf is just not a pleasant interface to work with, IMHO.

It's also memory intensive.

If you just want a reader, the thing that made it into breakpad is probably a
good reference (others may have better ones, i've thankfully been out of the
debug info game for years):

[https://chromium.googlesource.com/breakpad/breakpad/+/master...](https://chromium.googlesource.com/breakpad/breakpad/+/master/src/common/dwarf/)

It's mostly a callback interface, and has a function info reader meant as a
demonstration It should work without trouble outside of breakpad (when it was
contributed, I made it portable to be able to just compile standalone. it
doesn't look like much has changed).

it does not support DWARF4/5, but nobody should need to care.

It also has no expression evaluator but i have a bunch of them if someone
needs them :)

~~~
alschwalm
Nice writeup! I've done some work with libdwarf and I agree that it isn't very
pleasant. Do you have any suggestions for better interfaces that support
writing?

~~~
DannyBee
How much of a writer do you want?

Just something that writes DIE's?

Or do you need line info, accelerator tables, etc?

~~~
alschwalm
I think just DIE's would be sufficient, definitely don't need accelerator
table stuff.

I've actually considered looking more at the LLVM dwarf writing components. Do
you have any experience with that?

------
foota
If you haven't, take some time to look through the list of their blog posts at
[http://jvns.ca/](http://jvns.ca/) (yes I know you can click there from the
post but I'm trying to make it really easy) because they have written an
incredible amount of very interesting blog posts.

~~~
gedrap
Some of my personal favorites from her blog:

[http://jvns.ca/blog/2016/03/16/tcpdump-is-
amazing/](http://jvns.ca/blog/2016/03/16/tcpdump-is-amazing/) quick
introduction to TCP dump

[http://jvns.ca/blog/2014/09/27/how-does-sqlite-work-
part-1-p...](http://jvns.ca/blog/2014/09/27/how-does-sqlite-work-
part-1-pages/) diving into SQLite and sharing the findings with the readers

[http://jvns.ca/blog/2014/08/12/what-happens-if-you-write-
a-t...](http://jvns.ca/blog/2014/08/12/what-happens-if-you-write-a-tcp-stack-
in-python/) implementing TCP in Python

Most of them have valuable HN discussions as well.

~~~
tcprst
Thank you! Your comment prompted me to look closer at her blog. There's some
great stuff there!

------
userbinator
Every time I use GDB, I feel like it was almost deliberately designed to make
me hate using it, and those example outputs really show why; compare WinDBG's
output dumping bytes (and corresponding ASCII in the usual hexdump format,
something that GDB just doesn't seem to support at all...):

[http://3.bp.blogspot.com/-J5bsfRdkOdk/UsHqCho2huI/AAAAAAAACV...](http://3.bp.blogspot.com/-J5bsfRdkOdk/UsHqCho2huI/AAAAAAAACVI/4YGfzuzB3ds/s1600/screenshot.538.jpg)

and structures:

[https://msdnshared.blob.core.windows.net/media/TNBlogsFS/Blo...](https://msdnshared.blob.core.windows.net/media/TNBlogsFS/BlogFileStorage/blogs_msdn/ntdebugging/WindowsLiveWriter/WindbgTipKN.FrameDVandDTItssoeasy_848C/clip_image008_3.jpg)

~~~
noselasd
While I don't particularly like how gdb does this either, you can:

* "set print pretty on" , this makes structs readable

* "x/20b &var" to show a hex dump. But no ASCII, and you have to give it the count (e.g. 20 bytes..)

------
lazyant
Two gdb articles in a day! [http://www.brendangregg.com/blog/2016-08-09/gdb-
example-ncur...](http://www.brendangregg.com/blog/2016-08-09/gdb-example-
ncurses.html)

------
signa11
interesting to see folks like this :) btw, the paper "introduction to dwarf
debugging format" (available here:
[http://www.dwarfstd.org/doc/Debugging%20using%20DWARF.pdf](http://www.dwarfstd.org/doc/Debugging%20using%20DWARF.pdf))
is quite approachable. also, in case folks get carried away, there is always
the "linkers and loaders" book for in depth analysis.

edit-001 : minor formatting updates

------
philh
> If we want to find the address of a global variable in our program, all we
> need to do is look up the name of the variable in the symbol table, and then
> add that to the start of the range in /proc/whatever/maps, and we're done!

We just saw three ranges in ///maps:

    
    
        5598a9605000-5598a9886000 r-xp 00000000 [...]
        5598a9a86000-5598a9a8b000 r--p 00281000 [...]
        5598a9a8b000-5598a9a8d000 rw-p 00286000 [...]
    

what are these? The r??p look like permissions, read/write/execute/something,
but how do we know which one to look for the variable in?

(And what are the numbers afterwards? 00281000 on the second line is the
length of the range on the first line, but then there's a gap of 00200000
between the end of the first range and the start of the second. 00286000 on
the third line is again 00200000 less than the distance from the start of the
first range and the start of the second.)

~~~
teraflop
The man page for /proc (yes, there is one) explains what all those fields
mean.

In this case, each of those mappings corresponds to one of the sections in the
binary. The permissions indicate that the first one is executable code, the
second is read-only data, and the third is writable (copy-on-write). The
number after the permissions is an offset into the underlying file.

I suspect the article is glossing over some details e.g. how gdb figures out
which mapping corresponds to which section, but it gets the basic idea across.

~~~
pkaye
Wow in all these years using Linux I never thought of looking for the man page
for /proc. Lots of info there...

------
rurban
And the next time she will find out how gdb breakpoints work, int3.

Something like
[http://www.cs.columbia.edu/~junfeng/09sp-w4118/lectures/int3...](http://www.cs.columbia.edu/~junfeng/09sp-w4118/lectures/int3/int3.txt)

~~~
vardump
X86 debug registers (DR0-3, DR6, DR7) are also useful. They don't require code
changes. As an added bonus, you can set a breakpoint for a variable access,
which triggers whenever a certain variable is accessed.

Although unfortunately debug registers are limited to just 4 simultaneous
breakpoints in the same time.

[http://wiki.osdev.org/CPU_Registers_x86#Debug_Registers](http://wiki.osdev.org/CPU_Registers_x86#Debug_Registers)

------
sireat
I'd love to see a similar dissection of WinDBG, how it works and why.

------
pas
Speaking of GDB and jvns, how come there's no good out of process profiler for
Python?

[http://jvns.ca/blog/2016/06/12/a-weird-system-call-
process-v...](http://jvns.ca/blog/2016/06/12/a-weird-system-call-process-vm-
readv/)

I love the JVM's easily trace-ability, though that involves safepoints, so
that's not completely out of process either.

~~~
filereaper
Depends on how the language runtime defines its calling convention, does it
follow System Linkage [1] or does it implement its own internal linkage ie
Private Linkage?

If it follows standard System Linkage, its easy to point gdb or any other
system debugger or profiler to debug and profile the application.

Some runtimes have a mix of System and Private linkage, ie C functions will
follow System Linkage but JIT'ed code frames might follow private linkage.
This makes for difficult stack-walking by system native debuggers and
profilers. You'd have to teach GDB via an extension how to walk the non-
standard frames.

So yea, long story short, it depends on the linkage convention the
implementers of the language runtime decided to follow.

[1]
[http://www.x86-64.org/documentation/abi.pdf](http://www.x86-64.org/documentation/abi.pdf)

~~~
pas
I mean, GDB already has good scripts to extract CPython stuff from inferior
processes, but there's no GDB API so that anyone could make a profiler out of
it. At least I haven't found anything. (So GDB's scriptability seems to be
very restricted.)

------
ensiferum
It does? Well at least as long as you don't mention threads. Or try the cross
host debugging "experience".

------
Keyframe
I don't think I could ever use gdb without google or a cheatsheet at hand,
even after years of dealing with it! I'm not even ashamed because of it. It's
what it is.

------
fs111
yes, and "they" can be used to describe a single person:
[https://en.wikipedia.org/wiki/Singular_they](https://en.wikipedia.org/wiki/Singular_they)

~~~
ruraljuror
Yes it can be, and if that were the intention here (rather than a mistake) it
is a good example of poor usage. A sentence such as "Julia Evans writes on
their blog" introduces a lot of unnecessary ambiguity if "their" = "her".

~~~
chrisseaton
How are you supposed to find out the pronoun that someone prefers in order to
write that though?

~~~
PeCaN
You don't fucking care. You write ‘her’ if it's a girl and ‘him’ if it's a
boy, then if they ask for something different you apologize and correct
yourself. This isn't tumblr.

~~~
chrisseaton
Well exactly - you don't care and you just write 'their'. You're the one
suggesting that we should go out of our way to do detective work to find some
evidence of whether someone looks like a man or a woman. It's easier to just
say 'their'.

Also, I've seen several comments here of someone making what I presume are
honest mistakes in using the wrong pronoun and being publicly shamed for it.

~~~
ruraljuror
I don't know if it's necessarily easier to read, you are introducing a lot of
ambiguity. When you say "Julia's writing on their blog," it sounds like she's
writing on a blog that belongs to other people.

~~~
chrisseaton
> it sounds like she's writing on a blog that belongs to other people

It doesn't to me. Maybe it varies by region.

------
bogomipz
I have a question, why is it that every time this person/company updates their
blog with a new entry it ends up on Hackernews?

In my opinion this starts to feels very self self-promotional.

As someone who enjoys variety and diversity I think it's a valid question.

~~~
bogomipz
It's amusing to see the downvotes for asking a question.

This just reinforces my suspicion that that this blogger's friends and
coworkers are the reason a single blog constantly ends up on the front page of
HN regardless of merit.

~~~
DanBC
"Why is this here" questions are very boring and always get downvotes.

"I suspect vote rigging" accusations aren't nice. You should send them to mods
at the email address rather than post them to the thread.

