
How Does a Debugger Work? - wazari972
http://blog.0x972.info/?d=2014/11/13/10/40/50-how-does-a-debugger-work
======
cplease
Nice article, but it doesn't quite deliver. It says, the trick is not "black
magic", but then defines debugging in terms of ptrace syscalls, describing the
API a little bit, but without giving a clue as to how ptrace actually works.
So, ptrace is essentially black magic.

And this is not really an explanation of "how a debugger works," or even "how
gdb works." ptrace is just one of several debug targets for gdb. There are
simulators, core files, various embedded monitors, VxWorks, Windows, gdb
remote debug servers over various interfaces, and on and on. ptrace is
irrelevant to other targets.

~~~
planckscnst
I've been seriously thinking of starting a podcast that layer by layer removes
the "magic" behind Linux/Unix and even other OSs, even down to explaining how
processors/computers in general work. I'm wondering if people like you have
any interest in such a beast.

I have an episode ideas doc [1] for it that is several pages long. I need to
flesh out an episode and shoot a test. After that, I need to figure out
subject order.

[1]
[https://docs.google.com/document/d/1Nn01yDS5PkiegHq4Oxz43xUQ...](https://docs.google.com/document/d/1Nn01yDS5PkiegHq4Oxz43xUQRLHqbc5l5GCRjTEzRw8/edit?usp=sharing)

~~~
agumonkey
Gonna throw it just in case, I'm very fond of Massalin's thesis
[http://valerieaurora.org/synthesis/SynthesisOS/](http://valerieaurora.org/synthesis/SynthesisOS/)

A jitted, partially evaluation capable kernel. The thesis is very readable,
proof: I could enjoy almost all of it and I'm a monkey. It might give some
insights about kernel design that may not be told in mainstream ones.

ps: mandatory wikiwiki page
[http://c2.com/cgi/wiki?SynthesisOs](http://c2.com/cgi/wiki?SynthesisOs)

------
ChuckMcM
One of the more interesting things about the ARM Cortex-M series is that
debugging is "built in" to the CPU core on all licensed processors. No hacks
required. Something that I'm sure x86 machines would have had, if transistors
has been as cheap then as they are now. Of course early on Intel made even
_more_ margin on versions of the processor used for doing in circuit emulation
by 'bonding out' to an unused pad access to internal trace registers.

~~~
xxxyy
x86 does also have debugging registers:

[http://en.wikipedia.org/wiki/X86_debug_register](http://en.wikipedia.org/wiki/X86_debug_register)

They can be used from the Windows Kernel Debugger (WinDbg), although I'm not
sure if/how Linux-based gdb uses them.

~~~
MaulingMonkey
I'm pretty certain that Visual Studio uses those for "memory breakpoints", and
that GDB uses those for "hardware watchpoints", to use their respective
nomenclature. And these are _very_ handy at times.

------
msvan
I'm not too knowledgeable about this subject, but I've been interested in
learning how native code debuggers work for a long time. One thing I wonder
is, if the debugger inserts an invalid instruction or a hardware breakpoint
instruction into the code at runtime, wouldn't all of the in-memory code need
to be reallocated and recalculated in order to make room for the new
instruction and recalculate jump addresses? How is this handled?

~~~
yan
A breakpoint instruction is usually a single byte in variable length
instruction ISAs and the same width as any other instruction on fixed width
ISAs, so inserting a breakpoint is just overwriting the first byte of an
instruction and keeping all other offsets identical.

------
lostpixel
As someone who used to play with debugger implementations a bunch it's nice to
see some articles digging into this.

Only feedback I would give is to remove the shadow on your text, I had to
manually disable the shadow before I was able to read :).

~~~
wazari972
thanks,

regarding your feedback, I don't see any shadow in FF! The CSS is not mine,
and there's indeed a `text-shadow` set for the text so I've removed it. Let me
know if it's still bad, and maybe which browser you're using

------
jbn
Also relevant, and a good read to boot:
[http://www.cs.tufts.edu/~nr/pubs/retargetable-
abstract.html](http://www.cs.tufts.edu/~nr/pubs/retargetable-abstract.html)

------
agumonkey
Just `gdb gdb`. Might please infinite interpretation towers lovers around
here.

------
esfandia
Based on the title of the article, I expected it to describe very general
principles for writing debuggers, but it seems very specific to gdb. Are
things similar in, say, Python or Java?

~~~
chrisseaton
I wrote a paper about a very different way to debug Ruby
[http://www.chrisseaton.com/rubytruffle/set_trace_func/](http://www.chrisseaton.com/rubytruffle/set_trace_func/)
by AST rewriting and dynamic deoptimization.

------
MichaelGG
So how are the ptrace functions implemented? Is the "hack" of inserting
invalid instructions used even for single stepping? (Though hardware
breakpoints are probably easier?)

~~~
barrkel
Instructions that trigger a CPU interrupt, or other means of handing control
back to the OS, are used for single-stepping at the source level, yes.

I think "hack" is a little too strong a word. CPUs, historically, have had too
little state to usefully track all the execution breakpoints you might want to
set. So they normally have a nice short instruction that the OS can trap. On
x86, it's INT 3, aka interrupt 3, encoded in a single byte, 0xCC. Interrupts
are handled by entries in the interrupt vector table (IVT), normally in a
region of memory only the OS can write to (if you're not doing something like
running DOS in real mode). So when the CPU tries to execute 0xCC, it instead
calls code located in the fourth entry (3 counted from 0) in the IVT. This
will be filled by the OS, for modern values of OS.

(Single stepping CPU instructions is done with mode flags on the CPU for x86;
this let's the CPU single-step things like REPNZ prefix used with string
instructions).

------
wazari972
I've updated the article based on you comments, thanks :-) (title more
focused, How is Ptrace implemented, What about systems without Ptrace)

------
omegote
By the way, it'd be seriously cool to find a good tutorial about gdb. I've
been using it for years, but just the basic operations...

~~~
concernedctzn
Even just knowing the basic operations is a great starting point. One cool
thing I've been doing recently is using the python interface to script it for
those times when I'm trying to find something particularly hard to repro that
involves a lot of searching. Surprisingly easy to work with.

------
tcas
Don't modern processors support hardware breakpoints / watchpoints?

~~~
tryp
There is typically only one (or a small set) of hardware instruction pointer
breakpoints. Every time the CPU is about to execute an instruction, it
compares IP to the IP breakpoint register (or set of registers) and if they
match, a debug exception is thrown. These are what you must use to debug the
BIOS/bootloader executing in-place from ROM with a JTAG or XDP hardware
debugger attached. Later, software can add the proper exception handler once
enough of the hardware platform is configured. (Not all architectures exit
power-on reset with exceptions enabled or even exception vectors mapped to a
valid memory location.)

If the code you are debugging is executing out of write-able memory, then your
debugger can support an effectively infinite list of breakpoints by writing
some instruction that causes an exception to be thrown anywhere you want a
break, then handling that exception by looking up the faulting instruction
address in your breakpoint list.

Most processors also give you hardware data access breakpoints as well. These
typically sit on the CPU's data bus interface and can fire when the address or
data that's about to hit the bus match the respective breakpoint registers.
There is usually an option to only trap reads, writes, or both. Sometimes you
get interesting things like a mask register that lets you trap on a whole
block of memory.

One of the most interesting hardware debug features of modern processors is
the branch trace which keeps a running list of the last N branch instructions
that lets you reconstruct a "how did we get here" story.

~~~
MrBuddyCasino
Do you know if VMs like the JVM depend on hardware features also? I suspect
that would be more an optimization, not a necessity though, since they should
be able to de-optimize jitted native code on stack frames containing
breakpoints?

------
pmalynin
Didn't read the article (I find the topic rather bland), but debugging on
x86-64 is quite simple: you have your debugging registers (DR0-DR4) which set
trigger addresses and conditions (execute, read, write) and then call a system
interrupt when the condition is satisfied. This approach is limited to 3
breakpoints. Most moder debuggers do software breakpoints, that is when you
set a break point for a particular line or instruction, the debuggrr replaces
the first byte of the instruction with an int3 instruction (usually interrupt
instructions are two byte wide, so technically int3 and "int 3" are different)
but regardless the debugger slusually stores the actual instruction byte in a
table to replace the int3 when it is actually hit. I suppose one could do this
differently by causing a page fault (a simple bit switch from present to not
present in the page table) and then monitoring the CR2 register to get the
address of the executing code or the daya that is being accesed. One point I
forgot to mention is that the x86 has hardware support for single-stepping
instructions (a simple flag). But all of these methods require operating
system support.

