
“Give me 15 minutes and I'll change your view of GDB” [video] - DebugN
http://undo.io/resources/presentations/cppcon-2015-greg-law-give-me-15-minutes-ill-change/
======
derefnull
A summary of the 15 minute video:

1\. introduction to 'tui' mode in gdb
([https://sourceware.org/gdb/onlinedocs/gdb/TUI.html](https://sourceware.org/gdb/onlinedocs/gdb/TUI.html))

2\. intro to python support in gdb (if it was compiled in)

    
    
      2.1 python print('hello world')
    
      2.1 set breakpoints with python gdb interface
    

3\. Demo of reverse debugging in gdb, which helps isolate a stack-corrupting
bug
([https://www.gnu.org/software/gdb/news/reversible.html](https://www.gnu.org/software/gdb/news/reversible.html))

~~~
castratikron
GDB + Python is really slick. I wrote a Python C module for the first time
last month and GDB really came in handy for that. It just worked.

~~~
merraksh
Yes. It can print the value of C variables and Python objects, using their
_str() method I assume. Extremely useful. Even better when done within Emacs.

~~~
jamra
I'm more of a vim user, but have dabbled in emacs so I'm not sure what you
mean by "Even better when done in Emacs". Is there some kind of cool gdb tools
in Emacs you're referring to? I'm just asking because day by day I'm becoming
more interested in making the switch.

~~~
endgame
You might find this article useful:
[http://blog.aaronbieber.com/2015/05/24/from-vim-to-emacs-
in-...](http://blog.aaronbieber.com/2015/05/24/from-vim-to-emacs-in-fourteen-
days.html)

Aaron gave a great talk about his switch here:
[https://www.youtube.com/watch?v=JWD1Fpdd4Pc](https://www.youtube.com/watch?v=JWD1Fpdd4Pc)
, which caused several people at work to move off NeoVim and give emacs a go.

~~~
pjmlp
Thanks for the links it was interesting to watch as a former Emacs fan.

In the old days, Emacs alongside DDD was the only way for me to get some of
the Borland IDE comfort in UNIX land.

------
gizmo
A broken ncurses text gui debugger with broken navigation and byzantine
commands should somehow make us look more favorably upon gdb? We had better
debuggers than this in the early 80s. Turbo pascal had a decent debugger and
IDE that was right built in!

How come the open source world hasn't been able to create a debugger that's as
good as the (still primitive) debugger in Visual Studio 6.0? It's astonishing
to me how bad the state of the art is.

~~~
w0utert
>> _How come the open source world hasn 't been able to create a debugger
that's as good as the (still primitive) debugger in Visual Studio 6.0? It's
astonishing to me how bad the state of the art is._

I keep asking myself that same question every now and then, why can XCode have
such a nice and pleasant LLVM-based version of gdb, while gdb on linux still
doesn't have even the most bare-bones GUI that let's me inspect common data
types without jumping through hoops?

~~~
bitwize
What you are looking for is GNU DDD.

[https://www.gnu.org/software/ddd/](https://www.gnu.org/software/ddd/)

~~~
w0utert
I use ddd, but I don't particularly like it. It's telling that after all those
years we still don't have anything better, it seems like ddd today is still
the same crutch as when I first used it, over 10 years ago.

~~~
dman
The market has spoken in terms of preferring free tools. I personally think
that is one big reason why the quality of developer tools is as low as it is.

~~~
bronson
because free == low quality ??

~~~
alkonaut
For some reason the equality seems to hold for most applications with a
complex user interface. Reasons may include that it is a cross discipline
thing that requires skills/people such as interaction designers, artists,
technical writers, linguists etc that are less common than developers in the
OSS community. It could also be that good interface design benefits from a
centralized decision process which can be a bad fit for OSS that is usually
too incremental and democratic for radical design changes. Normally there is
no money for things like paid UI testing/focus groups.

That said, there is great free software with good UI/UX too - only they are
more or less always backed by a large corporation.

------
lucio
IMHO, In this day and age, a proper debugger should by default show source
code and execution point, should allow you to inspect variables, and change
its values, you should be able to manually set the executing position, and you
should be able to _alter code and re-compile on the fly_ , continuing the
debugging session.

I know this is really-really hard to accomplish, but some IDEs do it.

The only thing that's "optional" and a little over the top (but very useful)
is time-travel debugging.

~~~
vitd
When you say "alter code and re-compile on the fly," do you mean and continue
debugging without stopping the app and re-running? Because if so, that's a
terrible way to debug. You now have state that may not be possible to achieve
with the new binary you've made, and you may be debugging something that won't
every exist in reality. And it may be very hard to tell that's the case. That
doesn't sound very useful. It sounds very dangerous.

~~~
gnaritas
If you've never written code in a live debugger session, moved the execution
point back up, and immediately run over the code you just wrote, you have a
crappy debugger. VB6 could do this, Smalltalk does this, it's not dangerous,
it's damn productive.

~~~
cm3
How do you make sure there's no corrupted state and that the data structures
in memory fit the new code?

~~~
vvanders
You don't. Worst case is you had wrong assumptions and restart the program,
same as if you didn't have the ability to change things.

Best case thought, it's so useful. It's also incredible for editing tunables
on the fly(think videogame gameplay values, layout values for UI, etc).

~~~
qznc
Worst case is you don't know you broke consistency. Then you can waste hours
hunting for ghost bugs that only occur because of the hot-code-reload.

~~~
tedks
As someone who worked on dynamic updating research projects for years I can
attest to the really shitty bugs dynamic updating can cause.

------
catern
I prefer Emacs GUD over the built-in GDB TUI. I actually don't understand why
anyone would use the built-in TUI; you have to learn new things anyway, why
not learn the Emacs interface which not only looks way better but is more
reliable and has more features? The keyboard commands for the GDB TUI are even
(mostly) cloned from Emacs.

But as that person in the audience says, it looks like the presenter is just
not aware that the GDB TUI is a substandard version of Emacs GUD, having
mostly the same keybindings.

The second half is a good combined demo of some neat GDB features though.

~~~
Sanddancer
The biggest problem there is that you're injecting the politics of emacs into
your debugging then. GUD's hooks into LLDB are still a third party package
because certain parties have deemed llvm-based utilities politically
inconvenient.

[https://lists.gnu.org/archive/html/emacs-
devel/2015-02/msg00...](https://lists.gnu.org/archive/html/emacs-
devel/2015-02/msg00360.html)

~~~
catern
How is that relevant at all? LLDB is a completely different debugger. The fact
that I have to download an extra package to use LLDB doesn't affect my use of
GDB at all.

If LLDB eventually becomes as popular as GDB, I'm sure it will be included
into core Emacs then. But I have no interest in LLDB at the moment.

~~~
DannyBee
"I'm sure it will be included into core Emacs then"

No, actually, it won't :) RMS has said it won't, in fact.

~~~
catern
RMS is not the Emacs maintainer...

And even if he was, I don't believe he ever said "If LLDB successfully
replaces GDB as the most used libre debugger, even then we still shouldn't
support it in Emacs". He's not insane, you know.

~~~
GFK_of_xmaspast
On the other hand, the historical record:
[https://www.phoronix.com/scan.php?page=news_item&px=RMS-
Emac...](https://www.phoronix.com/scan.php?page=news_item&px=RMS-Emacs-Gud-
LLVM)

------
EliRivers
Is this entire set of comments suffering from people conflating the debugger
with an interface to the debugger? In much the same way that sometimes people
conflate their compiler with their IDE.

GDB is a fine debugger. I've never had any trouble with it at all, and I don't
recall ever needing a feature in it that it didn't support but other debuggers
did, although I'm not a very demanding user of the debugger; I rarely need
more than some conditional breakpoints and the ability to see the value of any
given variable or memory location.

~~~
qznc
If WinDbg were so much better than gdb, someone at Google would have
replicated it for Linux.

I'm fine with gdb and conditional breakpoints is usually the most advanced
feature I need. Only once I used bigger guns. Detecting the error meant run
into conditional breakpoint A, then enable and run into conditional breakpoint
B (which would have triggered a lot before A), and then inspect a variable.
The problem: The error only occured sometimes. Thus automate this via Python,
so gdb could auto-reexecute everything until it found the failure case again.

What I still miss is multi-process debugging. Gdb can only attach to a single
process, so you need multiple gdbs to watch multiple processes and this
introduces race conditions.

~~~
EliRivers
You've just reminded me; sometimes I need to run something under Valgrind, and
upon Valgrind not liking something, sparking a break using GDB. I expect
that's possible using other debuggers, but it's so, SO easy using Valgrind and
GDB. So by this reckoning, GDB is the best debugger I've ever used.

Bring it, WinDbg :)

------
vvanders
I get that he's being clever at the beginning by poking fun at Windows but
many of the shortcuts he talks about are available in Visual Studio and it has
a much saner set of defaults and initial information. Ask anyone who did XBox
and PS3 development which environment they preferred :).

The reverse debugging stuff is pretty cool though.

~~~
bitmapbrother
>Ask anyone who did XBox and PS3 development which environment they preferred
:)

I'm pretty sure if you asked anyone that did PS3/PS4 development they'll
probably say ProDG.

~~~
vvanders
I'll give you that PS3 had a better CPU profiler but ProDG was a mess compared
to Visual Studio. Crashes, disconnects, problems with source maps. If I
remember correctly it couldn't do data or conditional breakpoints.

Compared to VS and PIX it was night and day, although you probably already
knew that from my comment.

~~~
xigency
Really, they were both bad for PS3 debugging.

~~~
vvanders
Oh man, using VS for PS3? Yeah, I bet. I was mostly talking about how it
worked on X360 which is a bit better supported platform.

------
pag
I use UndoDB a lot. It's been increadibly helpful for debugging dynamic binary
translators, which in my case are x86-to-x86 JIT compilers.

I also use it for debugging unit tests for a static binary translator. When
I'm unit testing, I will set a brakepoint at the reporting of a failure, look
at how the native program and emulated program states diverged, then I'll use
reverse execution and data/code breakpoints to go back and forth to try to
figure out what went wrong. This is much better than having to restart each
time.

Over the years I've found many bugs in UndoDB and they've been pretty quick to
fix them. Their support is great.

I started using it as a student, on a non-commerical (i.e. free license), and
it was an essential purchase for my current work.

------
jey
There's an awesome free replay debugger GDB extension available for Linux that
most people don't seem to know about: [http://rr-project.org](http://rr-
project.org)

~~~
paperwork
The user interface to gdb is not very nice, otherwise a utility like RR looks
pretty amazing.

------
RJIb8RBYxzAMX9u
Developers don't use debuggers because neither the benefits of using one nor
the harm of _not_ using one are immediately obvious. And when you first start
learning how to program, the cognitive load of picking up a new language, its
tooling and quirks, you simply don't have the energy to understand yet another
piece of complex software. Worse, you may not even know there's such thing as
a debugger! So we all do "printf debugging" because printing to the screen is
the first thing you learn, and it's obvious how you can use it to inspect
program state.

However, this is a novice trap, as more experienced developers would know that
printf debugging may not work for subtle cases, and may even steer you in the
completely wrong direction. But by the time said novices encounter such bugs,
they've already picked up the "bad habit", and it's difficult to go back and
unlearn it.

Some also feels that needing a specialized tool to find bugs is a weakness, as
you ought to be able to reason about the code without aid. But that is folly,
for if so, why use editors with syntax coloring (shouldn't you know int is a
keyword w/o being pointed out) or auto-completion (shouldn't you memorize all
exported functions)? In fact it's the opposite: a good debugger greatly aids
in comprehension, since you get to catch corner cases happen in action, and
with sufficient context for you to figure out how it'd happened.

There was a time when many developers also don't use a D(VCS) of some sort,
for similar reasons. But of course git has a celebrity champion, and so the
rest followed. Almost all the excuses from this thread people had raised about
not wanting to learn how to use a debugger properly apply to git (and to all
D(VCS) to varying extents), but you'd be ridiculed for not using it. IMO
debuggers need a similar champion before our opinions change, but alas IIRC
Linus himself is not too fond of debuggers...

That said, the state OSS debuggers is terrible, so they're not helping their
own cause. Moreover, they are often "buggy" themselves. I put that in quotes
because sometimes a "bug" is actually correct, but unintuitive behavior for
inexperienced users (pop quiz: how do breakpoints work?), or just poor /
nonexistent documentation. Nonetheless, the result is the same, and nothing
loses your trust in a tool faster than finding bugs in it while trying to fix
your own bugs.

------
hitr
I don't compare gdb to visual studio but to windbg.It would have been great if
gdb has support for debugger extension s like windbg(or any windows debugging
tools like cdb or ntsd etc). I can easily debug any .net programs on windbg
and level of details sos,psscor or many other extensions show you on for .net
debugging is amazing. There are windbg extensions where I can write SQL like
queries on .net memory objects or write python scripts or JavaScript to
control your debugger or write automated analysis very easily.I have seen the
same for php and nodejs processes with windbg and it was not very difficult.On
Linux the story is different and there is no easy replacement .When Microsoft
ported .net to linux with dotnet core ,they chose lldb instead of gdb as gdb
does not provide extensions support.I also heard that lldb is not as good or
stable as gdb.

------
danilocesar
I think this a good prelude for a good talk. I'm wondering if the guy is going
to release the full talk he gave later...

However, mentioning advanced-gdb usage nowadays and not mentioning RR
([http://rr-project.org/](http://rr-project.org/)) is a sin =)

------
dahart
TUI mode is cool, I didn't know about that!

Missing from this video, but the most powerful part of using gdb for me
recently has been the ability to do procedural debugging. You can easily write
debug scripts that are re-runnable instead of reproducing bugs manually. You
can sequence together multiple conditional breakpoints, keep debug variables &
state, and other stuff that is really tricky to do in more common GUI based
debuggers. I haven't used Visual Studio in the last few years, so maybe
they've added it. I had tried for years and never found it, resorted to gdb
for the hard problems, when I could.

------
ngneer
What I never liked about gdb was that I could not really figure out a way for
it to show me what has changed when stepping over an instruction or a call, a
feature that I often find very useful in other debuggers (especially in x64dbg
and edb if you do not have the source).

The debuggers I have encountered seem a bit limited when regarding the
debugger as a means of querying a subset of program executions (out of all
possible executions), but they are usually helpful in picturing system
dynamics. It has always been very helpful for me to allocate a portion of
fixed screen real estate to memory, variables and registers that I am watching
and see the changes as I step through, allowing me to observe the dynamics of
the system.

Too bad I could not figure out a way for gdb to help with that, the closest I
got was examining the memory I was interested in ('x' variants) before and
after, but no highlighting. To be fair, it is good at other things though
:)...

------
corysama
More recent talk from the same presenter
[http://undo.io/resources/presentations/accu-2016-become-
gdb-...](http://undo.io/resources/presentations/accu-2016-become-gdb-power-
user/)

------
dicroce
My favorite gdb feature? write watches. Set a write watch on a variable (even
by address) and get a break point when the variable is written to. Read
watches work the same way but break when a thread reads the variable.

------
chuckcode
Reverse debugging is a nice feature but just skip to 9min in to watch the demo
of reverse debugging. Surprised to see so much of the video talking about the
UI of gdb which certainly didn't change my opinion of it. I've found emacs+gdb
to be very useful over the years especially when debugging on remote servers
but this demo just reinforced my opinion of gdb being a very powerful program
that is pretty difficult to master. Also reinforced my opinion that you should
open your talk with at least some of your best content rather than saving it
for the last few minutes.

------
the_mitsuhiko
I'm glad LLDB exists now. It brought competition into the abysmal Open Source
debugger space. It's just a pity that there are no good frontends for LLDB so
far.

~~~
pjmlp
From the WWDC talks XCode looks quite close to VS experience.

~~~
to3m
Xcode does work, mostly, as you'd expect, since it's backed by lldb. And it's
certainly more like VS than command line lldb is! But still, the UX isn't
quite there. I found it rather limiting and fiddly to use - you don't get good
control over where the various windows go, and too many things are forced to
share a window. You can only see one of such things at once, and have to do
something to switch to the other.

This criticism applies to the UI overall, which puts things in inconvenient
places (the tall, narrow find results/compile results panel was particularly
ridiculous), and then won't even let you move them, but I found it especially
tiresome when debugging. This is the last thing you want in a debugger UI in
my view! I want to see as much stuff at once as possible, and I don't want to
have to switch between tabs unless I deliberately set things up that way
myself.

There's no separate registers window (it shares its window with the locals),
there's only one watch window, and the disassembly window never seemed to work
very well (when stepping, execution still proceeds on a line by line basis). I
recall the disassembly window also sharing a window with the source window
too. I think the memory view might do that too. All in all, most
unsatisfactory.

Visual Studio does have some odd limitations and annoyances, but you can at
least put the windows where you like.

------
abc_lisper
GDB tui is old. I remember using it 11 years ago, debugging a VM for Itanium.
Also, if you know emacs, gdb emacs integration will blow your mind.

------
kelvin0
Looks like VS Code has a plugin that integrates with GDB.

[https://blogs.msdn.microsoft.com/vcblog/2015/11/18/announcin...](https://blogs.msdn.microsoft.com/vcblog/2015/11/18/announcing-
the-vs-gdb-debugger-extension/)

Maybe polishing that plugin could go a long way in a better dev experience
than the current GDB UI?

~~~
mastax
This is VS 2015, not VS Code. Though VS Code does have some C/C++ plugins
that, I assume, integrate with GDB.

~~~
kelvin0
Well I stand corrected. I also have an open source project to start it seems
:)

------
mathgenius
Who writes bugs that need a reversible db to find? or an embedded python
interpreter in your debugger? I've done some hairy things in gdb to find bugs
(actually valgrind is a big help with this aswell) but these other features,
yikes..! If i ever need them it's gonna be a dark day for sure.

~~~
ethan_g
Well, "need" is a strong word. The features are definitely helpful though.

With large programs, there can be a really long distance between where
something is created/modified and where it's used. If there's a bug you catch
that occurs where the object is used, reversible debuggers help you trace back
to where it's created, or who modified it last.

If you're at a place that's hard to put a useful breakpoint (e.g. a widely
used function), and need to answer a what-if question, being able to
manipulate state from the debugger is super useful. You might not be able to
easily insert code to the program to create that state without messing up
everything else (again, example is a widely used function, where inserting
code could mess up other users of the function).

------
DebugN
Just to let you know that the same guy is running a series of 15 minute
webinars this week on how to become a GDB pro. First one is today (4pm
London). [http://undo.io/become-gdb-power-user/](http://undo.io/become-gdb-
power-user/)

------
nightcracker
I think GDB is fine architecturally for debugging.

One day maybe we'll see a good non-buggy GUI frontend for GDB that shows code,
local variables, assembly, etc in separate windows, all clickable to change
values, set breakpoints, change interpretations, etc.

------
DebugN
The same guy is giving a series of 15 minute talks this week via webcast
(first one is tomorrow at 4 pm London time)

[http://undo.io/become-gdb-power-user/](http://undo.io/become-gdb-power-user/)

------
dimdimdim
Here is a free video course on GDB - covers most of things discussed in this
video and more:

[http://www.pentesteracademy.com/course?id=4](http://www.pentesteracademy.com/course?id=4)

------
rosstex
"Sorry to the Windows users out there"

Not anymore! My, how times have changed.

------
jhallenworld
Interesting! I used to use in circuit emulators with trace buffers. You could
use them to discover that an interrupt handler messed something up by looking
back in time.

------
protomok
Wow, I didn't know gdb has built in reverse debugging or the Python
interpreter. The reverse debugger looks awesome!

I'm curious what people use the Python interpreter for?

------
gshx
Reminded me of all the hours spent debugging with dbx. Turns out, even after
all the yrs, they are still quite similar.

------
akp__
b _exit.c: 32

This guy is my hero.

------
johanneskanybal
Nice troll brain

