Hacker News new | comments | show | ask | jobs | submit login
Give me 15 minutes and I'll change your view of GDB (2015) [video] (youtube.com)
314 points by pmoriarty 6 months ago | hide | past | web | favorite | 92 comments

I used to think GDB was a tool with the most broken interface I've ever seen, and which requires arcane commands to do the most trivial of debugging things.

I still do, but I used to too.

That early dig against Windows was particularly funny. There's no way I would pick that over Visual Studio's debugging tools.

I agree that gdb's syntax is ridiculous, having come from a background of DOS DEBUG and WinDbg, but what irritates me more are the implementations of certain functionality:

- No way to get a 16-column (bytes+ASCII) standard hexdump. This is functionality that even the most basic debugger should have, yet it's missing from gdb.

- "disassemble" command is next to useless.

- If you write "b 0x12345" intending to set a breakpoint at address 0x12345, it doesn't work. An unnecessary and nonsensical extra asterisk is needed (which makes it look like it's retrieving 4/8 bytes from 0x12345, and using that as the address of the breakpoint.)

- Starting gdb with a binary and passing arguments to it --- you'd expect it to be smart enough to realise that anything after the executable name should be the arguments to the debuggee and not the debugger, but it isnt.

I don't use gdb often, but when I do, it's usually the option of last resort.

First of all, I agree! GDB without extensions is not that nice to use. But there exist several extensions (https://github.com/cyrus-and/gdb-dashboard, https://github.com/hugsy/gef, https://github.com/pwndbg/pwndbg, https://github.com/longld/peda, even though developed for it, these are not only useful for exploit dev!) that make it a lot better.

> - No way to get a 16-column (bytes+ASCII) standard hexdump. This is functionality that even the most basic debugger should have, yet it's missing from gdb.

This should be quite easy to implement as a small python snippet that you put in your ~/.gdbinit. Granted, this should be builtin functionality, but it also shows how extensible GDB can be.

> - If you write "b 0x12345" intending to set a breakpoint at address 0x12345, it doesn't work. An unnecessary and nonsensical extra asterisk is needed (which makes it look like it's retrieving 4/8 bytes from 0x12345, and using that as the address of the breakpoint.)

Agreed, I never understood why that was necessary. It is really annoying. Perhaps someone wanted to avoid a conflict if you had a symbol named "0x12345"? (can you even do that?)

> - Starting gdb with a binary and passing arguments to it --- you'd expect it to be smart enough to realise that anything after the executable name should be the arguments to the debuggee and not the debugger, but it isnt.

I recently learned that gdb has a neat option called --args, that does exactly this:

    gdb --args program arg1 arg2
GDB feels quite similar to VIM to me: you have to spent some time to get used to it, it has some warts but it is very flexible and useful if you know it.

"b 0x12345" will place a line breakpoint at line number 0x12345 of the current source file. The "nonsensical" extra asterisk is used to tell gdb that you want an address breakpoint instead.

Oh! That makes sense, thank you. It only raises the question though, does anyone really use hexadecimal line numbers? :)

Just a guess: if you have an expression language that lexes numbers in a few bases, then when parsing some commands you no longer know which base the number token was, all you know is that you got a number, and in case of the break command you interpret that as a line number unless prefixed with a * token

That's about the same vibe I got from GDB so far. I don't use it all to often, so I'm certainly not that "effective" in it, but I still get my debugging done. And yes, it is a bit arcane.

However, on the topic of picking this over the VS debugger: The lack of arcane configuration, socket permissions and usage steps required to get it running on remote machines is what would make me pick it over VS nearly all the time. I haven't yet gotten to try the VS2017 one, but its predecessors were abysmal in that regard.

Tellingly, GDB's simplicity in that regard - which is the same as far as many other UNIXy tools go - comes from the approach of "if you have an SSH connection, you're good to go", which is something that is lacking in almost any Microsoft tool.

GDB is used in lots of places, often times called programmatically. You can't just change long-standing behaviour and get away with it. This is not a web framework.

So additional positional arguments on the gdb command line will always be interpreted as core files or PIDs instead of being passed through to the inferior program (unless using --args).

Similar things apply to other complaints.

You don't explain how "disassemble" is useless, but the rest of the things you mention are extremely superficial.

I could agree with you on gdb's user interface. However, working on embedded hardware, plain gdb and it's architecture-specific forks|variants have been the least fiddly things to use.

The most recent examples I've stumbled upon were System Workbench for STM32 and Atollic Studio debugging relatively recent STM32F7 series. When the debugging from IDE doesn't work, it's very hard to see where the problem is (is the hardware interface service started? with the correct parameters? does it have access to hardware USB device? did it even succeed flashing the firmware?)

I stopped using the integrated debugging from an IDE because of the problems you described. Segger Ozone is doing a good job if you like a graphic interface.

(BTW nice to see you here again. I think we met at some point playing with a laser cutter next to your old office in Riga).

I've accustomed myself to openocd, gdb and some small custom scripts (e.g. interpreting fault registers)

Hehe, the world really is that small :) Say hi to Mark if you meet him, I haven't seen him in ages.

GDB already felt like that when we had C++ Builder, Visual C++, Zortech C++ visual debuggers on mid-90's versions of Mac and Windows.

The only debuggers that could match them in usability and graphical debugging features were Motif based ones, being sold as a single product.

From a historical-software/UI perspective, I'm curious what that Motif software looks like, and how, er, findable it is.

Studying old systems that have fallen by the wayside can often be instructive.

Still found on Aix and HP-UX systems, which are the surviving ones. Solaris dropped it, and all other commercial big iron UNIXes are dead.

You can get some screenshots here



Woops, didn't notice this till now. I wasn't referring to Motif itself, rather the debugger programs. Contemporary UIs are always interesting to study!

It's awesome that Motif and CDE are finally open source though.

There are convenient graphical frontends for GDB. DDD for instance can create plots, diagrams, etc.


ConqueGDB in vim is my jam.

DDD was always for me the only sane way to use GDB. Until I got hold of it, I was mostly using it from Emacs.

For what it's worth, VSCode and Visual Studio now have GDB integrations through its machine interface (MI). I use the VSCode one (part of the c/c++ extension) on a regular basis and while lacking functionality, it's very easy to use.

In GNU Emacs one can use gdb-many-windows in gdb debug mode to get the real power of gdb.

Can you tell me some of the things that VS does much better? I've only ever used GDB, and I am kind of lost when trying to debug a binary on windows

> I am kind of lost when trying to debug a binary on windows

When I want to debug a program using the Visual Studio IDE I do the following:

1. Start the IDE and open the solution file.

2. Make sure the Debug Build option is selected.

3. Open a given file in the solution that I want to debug.

4. Select a line in that file and use the Debug, Toggle Breakpoint menu to set a breakpoint.

5. Run the debugger using the Debug, Start Debugging menu and wait for the breakpoint to be hit.

6. When that break point is hit just use the debugger to examine the state of the program.

While a simple series of steps it's still pretty complex compares to the equivalent in gdb:

  $ gdb ./binary
  (gdb) break file.c:line
  (gdb) run
I think this is merely a disguised version of the "CLIs are too hard" argument...

Calling it "CLIs are hard" is trivializing it but anyway, it depends on what your common case is. For C/C++, CLIs slow me down tremendously. In an IDE, You don't need to type a single command to view program state (automatic + local + thread storage), call stacks, memory usage, CPU Usage, Heap activity, unpacked C++ containers, GPU state, etc. But this is not even that impressive TBH. FME Windows in general is better at debug symbol management than Linux especially when it comes to shipped binaries. The post[0] sums it up nicely:

"I can take a crash dump file from a customer running an arbitrary version of our games, running on an arbitrary version of Windows, load it into the debugger and have all of the symbols automatically show up. I don’t need to know or care what version of the game or what service pack of Windows the customer was running, and I don’t need to think about package names or new repositories. The symbols and binaries Just. Show. Up."


The symbol thing sounds useful and is something I cannot comment on, since most often I either debug open source programs that I run myself (so I have the binary with compiled with debug symbols) or there is no source at all for the binary (and so no symbols). But, most of the other things you listed are, IMO, quite easy with GDB:

> view program state

in my experience, your often only interested in a few variables. then, you just run

    display var
once for each variable and GDB will show the value of var on every break. There is also `info locals` to show a local variables, and I am pretty sure that you can set it up to run on every break with some simple config (there's probably an extension pack that does it)

> call stacks

almost every enhancement plugin for GDB in python that I know of does this (GEF, voltron, ...)

> unpacked C++ containers

I am pretty sure GDB pretty-prints C++ containers?

This isn't to defend GDB, it cannot do Heap activity or CPU usage or GPU state out of the box and sometimes, a visual interface is nicer. I just wanted to comment on what is possible with GDB, because I was often not aware of that either.

>This isn't to defend GDB, it cannot do Heap activity or CPU usage or GPU state out of the box and sometimes, a visual interface is nicer. I just wanted to comment on what is possible with GDB, because I was often not aware of that either.

Surely then, the difference between whats possible, and whats easy/default out of the box isn't simply "CLIs are hard" ! :)

Anyway it seems like we'd agree on most things, so nothing really left to discuss.

However, it is a lot more intuitive compared to gdb: Click the green arrow. Click the line you want to set a breakpoint on. See where you stopped in the same editor that you wrote the code in. Hover your mouse over variables to see their values, visually explore class and array variables. Dig down into functions that are about to be called, before you actually call them.

> While a simple series of steps it's still pretty complex compares to the equivalent in gdb:

Oh come on. If you're developing with visual studio, you've already got it open and have been compiling with it.

Debugging in VS is literally click the mouse where you want to stop, F5, full call stack/local variables there in front of you.

Given the list above, if you reproduce a similar list for gdb it is also pretty tedious: run a terminal emulator, spawn a new shell, cd to the directory you want to work in, compile the program with gcc with the appropriate flags, run gdb with the appropriate arguments.

Visual Studio has a command window that you can use if you want to type commands rather than use the GUI. Plus, the person you replied to was being very descriptive. Just as you start gdb and pass the binary name as a parameter, you can do the same in Windows by double clicking on the solution file.

Now try complex logic, only break here if this this and that happened just before. GDB can be much better for that.

Where VS shines for me is inspecting the state of datastructures. As long as you don't need to do any complex search on them, or programmatically walk a graph of them.

> a binary

I'm not sure that there's a VS project and source code available for the binaries in question.

Well if there isn't, you can start the EXE and do Debug|Attach from VS. Or (I think?), you can go to File|Open Solution, open the EXE directly, and let VS create a temporary solution for you that will let you run the EXE as a debuggee by pressing F5.

If you're debugging an EXE without source because you're writing a plugin or addon for it as a DLL, and you want todebug that DLL, you can specify the EXE in question as the EXE to run when you start debugging as part of the properties for the project for the DL in question. Visual Studio will spot when your DLL loads, load the debug info for it, and activate any breakpoints and so on. Works well.

(If your addon runs as a child process, you need the Child Process Debugging tool. Transformative. Get it here: https://marketplace.visualstudio.com/items?itemName=GreggMis...)

When you don't have source code, VS is not the best environment ever - it's oriented very much towards worknig with C/C++/etc., and there's a few options you have to play with to get it to show you disassembly and so on. But it's adequate. (One thing I like about it very much is that the step commands are the same whether you're stepping through source code or whether you're stepping in the disassembly window. I'll put up with quite a lot in exchange for that.)

In Visual Studio 2017 you can open an executable directly and begin debugging it without having either of those things.

So... just like every IDE that integrates GDB then ?

Exactly. gdb is just like gcc. I can see the point of having a command line version of a compiler for toying and for automation/build scripts etc, but I rarely see the need to type out compiler commands. Same with a debugger.

Debugging a binary that you don't have source to, and that you don't intend to modify the reduces the benefit of the full IDE (where your syntax view in debugging is the same as your editor). Still, you get all the niceties of the IDE. You break on a line, step further on some lines. Notice the bug happen after y = foo() + bar(). So you wonder "where are those?" and you F12 with the cursor on the method name to nav to that method once there you set a breakpoint on the first line, and run so you debug that next.

Also, if you add in edit-and-continue so you can modify the code and have it automatically recompiled and keep debugging it, that adds value too.

I assume that various editor adaptations for vim/emacs etc means that you get almost all the IDE experience with gdb as well (symbol navigation, syntax highlight etc) and at that point there is of course little difference between one IDE and another.

you have plenty of graphical frontends that bring it "down" to the ease of use of VS. I like QtCreator's gdb integration personnally.

> That early dig against Windows was particularly funny. There's no way I would pick that over Visual Studio's debugging tools.

Throw in the Sysinternals suite and hopefully powershell continues to improve and you have a very nice debugging/sys admin/etc toolset.

I just wished windows had the elegance of the unix-style file system and was as open as linux/bsd/etc.

What is at all elegant in UNIX style file systems?!

Windows is as open as OS X, HP-UX, Aix, Solaris, OS/400, ....

>I still do, but I used to too.

I see what you did there. - Mitch Hedberg

If you are curious as to why you're being downvoted, it's because this comment adds no substance and states something completely obvious.

In other words, it was unnecessary. That kind of stuff is for reddit.

No, I am not curious. I know exactly why I am being downvoted.

HN'ers sometimes need to lighten up a bit. A little humor among thousands of serious comments (many not useful) is not going to bring down HN and it is incorrect to assume it doesn't add any substance. My comment was not crass, sarcastic, degrading or mean. It was appreciating the commenter's sense of humor in a harmless way.

There was no humorous value in your comment.

People who are familiar with Mitch Hedberg's work don't need any help in recognizing the quote.

For people who haven't heard about Mitch Hedberg, your comment doesn't make any sense without further research.

To me it sounded like you're bragging that you understood the reference.

The comment tells you to go look up Mitch Hedberg, that's much more useful than further explanation since explanation is the death of humor.


This is another example where HN needs tags.

The original humorous comment could have been tagged "humor" and the parent reply could have been tagged "meta", and an HN reader could have their client filter posts based on their preferences: if they want to see humor or meta posts, they could have them -- if they don't they won't and their feed will be leaner and less annoying to them, and there'll be less need to downvote "off-topic" posts.

Then everyone would be happy.

I really like this idea.

[meta], [general-agreement], [non-contributing]

Until people spam the non-contributing tag on posts they don't like in an attempt to silence them, and companies abuse it to try and cover up posts that badmouth them, giving mods a constant headache.


If you are curious about why you are totally wrong, his comment was NOT unnecessary, and it DID add substance.

I had never heard of Mitch Hedberg, and therefore did not get the quote, and now I have enjoyed several clips of his comedy routines.

I agree with the pthreads, many people on HN are far too quick to hit the "that stuff is for reddit" button, and could definitely lighten up a bit.

Just wanted to mention gdbgui, a new front end to gdb that I developed which is similar to Chrome's debugger, has a gdb terminal which you can fall back to if you want to run gdb commands, and data structure/value plotting and visualizations similar to ddd.


Awesome! Does it support gdb startup scripts? I.e. the gdb -x option which instructs gdb to read and execute commands from a file and then drops you off in the normal prompt. I regularly use that, because it lets me keep and organise startup options for the programs I work on. I really don't want to have to type all the "set environment" &c stuff that's necessary for the programs I want to debug.

Does it (can it) work with the gdb pretty-print API? On the console I can define pretty printers for my data types using python, so I can examine them easily. I almost never want to explore a struct by expanding its fields, I just have a bunch of python scripts that extract a summary of what's important. E.g. for a 3-component vector type I don't need or want something like

    x = ...
    y = ...
    z = ...
I just have a pretty-printer that spits out "MY::Vector(x, y, z)". Similarly for a whole bunch of other classes. I especially don't want to look at STL objects by field, that's just insane.

Honestly, after a couple of years of using GDB, the console interface isn't really a problem for me, and I've become quite attached to the many customisation and automation features it offers. While the commands and syntax are all kinds of gnarly, I don't think I can ever go back to a debugger where I have to do everything manually via direct control. Better, faster, more intuitive direct control is always a good thing, but I don't want it if it's at the expense of the programmability of the debugger.

This looks great! Did you know that you can also use Chrome's debugger to debug Node? I wonder how difficult it would be to use Chrome's debugger as a front end for gdb.

Never tried, but pygdbmi would probably get you halfway there. It uses python to parse structured data (key/value pairs) from gdb output. Gdbgui uses it on its backend. (Disclaimer: I am the developer)


Also for golang: https://github.com/cyrus-and/gdb

If you want to use a powerful gdb visual interface, you may want to try DDD. However, DDD has an old-fashioned GUI which is very crude. I have tried several visual debugging tools, such as RedHat Insight, DDD, Nemiver and kdbg, but none of them is enough powerful or stable. As far as I know, DDD can't display non-English comments in source code.

Now I'm using an awesome GDB init file to construct a convenient debugging interface. It's an open source project and you won't regret it: https://github.com/cyrus-and/gdb-dashboard

There is also a browser-based frontend for GDB: https://github.com/cs01/gdbgui

If you want to try more, search the keyword GDB on GitHub and sort by stars, then you will find many GDB enhancement tools.

Qt Creator is also a fairly decent front-end to gdb ( http://doc.qt.io/qtcreator/index.html ).

Looks interesting. Now, if only I could get combined manual/automatic debugging in CDT.

Previous discussion from a year ago: https://news.ycombinator.com/item?id=12101347 (169 comments)

Some other very nice:



I particularly like Voltro's view modularity and backends support. But dashboard is a really nice throwback (SoftICE) with its theme and sensible default layout.

GEF as well: https://github.com/hugsy/gef

Voltron adds awesome views, GEF adds a lot of handy functionality. I use both (often together)

Voltron seems interesting, but on the first application I tried it on, it caused my gdb to hang, and the views remained blank... Hopefully it'll mature fast.

If you want a powerful debugger with a crappy interface that is widespread (and programming java) try jdb. It comes with the freaking jdk tool chain so it is basically everywhere, however, gdb can't hold a candle to the horrible experience it is to use jdb without a graphical frontend. The documentation is about a page and a half, that is all, classes must be fully qualified with packages (which can have dozens of levels) in all commands, and it does not use readline for input!!! No left or right arrow to move the cursor! /rant

The TUI in gdb is a great way to get started with gdb. Instead of writing your own hooks and commands you can visually step through the program and source code ( if you have debug symbols enabled)

I think that more people should know about this feature.

Absolutely. Many of the comments here deride the default gdb interface but the TUI is orders of magnitude more usable and I only learned of it recently. (I much prefer it over Emacs' interface, for example, something another commenter recommended.) And in single-key mode it's even better.

I found rr to be much better for reverse debugging than gdb's record feature. For some reason, hardware watchpoints while reverse-continuing don't work for me when using gdb record but do when using rr.

He mentions this in his longer talk:


There are only three comments so far so small sample size but am I the only one who actually likes working in GDB?

Then again, I also use Vim as an IDE. But so do a lot of folks here.

for those who are still prefer documents over video, pdf slides are at https://github.com/CppCon/CppCon2016/tree/master/Tutorials/G...

radare2 recently added reversible debugging feature too: https://radare.gitbooks.io/radare2book/content/debugger/revd...

Note, that you will need version from git, but anyway, new release of radare2 will be ready in less than a couple weeks.

The "flipping back and forth" that he seems so impressed with, combined with the arcane key bindings (I know they're Emacs-inspired, but that's little help when the navigation model doesn't actually work like Emacs windows/buffers), is why I dropped TUI pretty quickly. Besides, I have 3 1080p monitors; a UI designed for a VT220 is not the best use of that setup.

It also doesn't help that I've never gotten gdbserver to actually work reliably on my embedded Linux targets (which, to be fair, are not Cortex-A/Apple/Qualcomm chips, so probably being tested by ~nobody). I'd understand if some operations aren't supported due to missing debug hardware or emulation magic, but stuff tends to just start mysteriously breaking (e.g. not stopping on breakpoints, hanging when trying to print memory, crashing on resume/step, etc.).

If don't like the Ctrl-X,A shortcut to get into TUI mode there is a command line option (--tui) to get GDB to start up in TUI mode. I find that a lot easier to remember.

I think cgdb (https://cgdb.github.io/) is even better than TUI mode though. Some nice things that it has that TUI mode doesn't:

- Colorization of the source code window

- The terminal display doesn't get confused when the debugged program prints to stdout

- You can still use the up and down arrow keys for moving through your command history and don't have to resort to Ctrl-P and Ctrl-N. Scrolling the source code window also uses the up and down arrow keys but you have to switch the "focus" to the source code window first.

> - You can still use the up and down arrow keys for moving through your command history and don't have to resort to Ctrl-P and Ctrl-N. Scrolling the source code window also uses the up and down arrow keys but you have to switch the "focus" to the source code window first.

Minor correction, that's the same as gdb tui. Use ^X,O to select which window has focus and you can use up and down arrow to go through your command history. The rest is definitely better in cgdb.

As I've written on this subject before, I think current debuggers largely miss the point, though gdb misses the point in a whole different league.

What I want from a debugger is 99% of the time met by gdb by running a program and running a backtrace and inspecting a few variables. Where gdb falls flat by default is in making me type in commands to get a register view, backtrace, disassembly of the current area, and the current stackframe.

Most other debuggers do ok there. But even with gdb it's trivial to define a few macros to make that part easy.

Where every debugger I've used falls flat is that for most harder problems, their functionality seems focused on heaping on functionality to let me do the job the debugger should have, instead of actually doing the things that would really save me time.

I've increasingly instead turned to valgrind as my first choice, for the simple reason that a huge portion of the problems I deal with are low level (e.g. debugging my compiler, which involves debugging things like why my code generation doesn't correctly initialise a value on the stack. But this is the same problem I generally would like to be able to debug in higher level languages:

Program crashes. Which value caused the crash, or caused the exception to be thrown? When was that value set? Where? What was the code paths that led to that value being set? Which variables contributed to it being set to that value?

Fully automating this in the general case is not possible (it devolves to the halting problem if you don't put the debugged program in a "black box"), but making it easy for a human to guide it there should be the priority for a debugger.

Being able to get a quick view, see that "huh, that value shouldn't be there", be able to tell the debugger "identify when that variable will change and re-run", and work your way back that way, would solve the vast majority of the tough debugging sessions I've had.

Note that the above is not the same as just setting watch-points by tracing and checking. Handling of watches is usually crude in most debuggers. What I'd like is something similar to Valgrind - to do this well the debugger needs to at least analyse the code and intelligently pare down what it watches, and ideally dynamically translate the code into something sufficiently instrumented to let it run fast. For most debugging sessions where I've tried watches, I've had to give up because of how slow they can be, but if the issue is something valgrind can catch, it finds it fast.

Support for replays might be an improvement, but it's still about providing tools for us to do the debuggers job instead of semi-automating vital functionality.

You should try using rr. The combination of reverse execution and hardware data watchpoints enables new, much more direct, debugging strategies to answer the kinds of questions you posed.

rr doesn't work on my CPU yet. Or at least not without building from git, and that fails due to some compilation problem or other. I haven't had time to try much more than that, but from what I can tell from the docs it does seem better though I think at unnecessarily high cost. I already have code that make Valgrind fall over because its state exhausts my memory, so I'l be interested in seeing how rr fares there.

File an rr issue if you have trouble getting it working.

The reverse debugging portion of this video was mind blowing. I had no idea you could do that.

WinGDB exists and allows remote debugging in Visual Studio to remote machines using gdbserver. Used that in my previous job and was very convenient.

A video targeted towards people who haven't spent fifteen minutes in gdb but already think it sucks. Cool beans, just what HN needs

I prefer valgrind over GDB.


15 minutes!?! Gosh, even changing my views will be slow with GDB ;-)

I'll take a textual article over video any time.

Agree. I started embedded coding on Linux + Qt-Creator. I have everything working, compiling and running on my ARM and AVR chips, but Debuging jsut does nto work. Just can't find any proper info on how to integrate GDB into Qt-Creator and be able to just Debug like in Visual Studio. If this is not possible, that this endeed is very stupid, starting/stopping servers on a remote target and the rest...

Looks like you need to talk to nocman https://news.ycombinator.com/item?id=14999140 or jcelerier https://news.ycombinator.com/item?id=14999140?

Assuming http://doc.qt.io/qtcreator/creator-debugging.html (which someone else noted) didn't help.

I don't think the situation is likely to improve much unless as a profession we become open to paying for quality tools.

We used to be open to paying for quality tools, then the FOSS generation took over, killing the business of selling tools to single developers.

Nowadays the only way to make money selling developer tools is to sell them to enterprises, the only ones that value developer time vs money spent on tooling.

Why not use cgdb?

I much prefer the cgdb interface to gdb's tui. I've also used it with various embedded builds of gdb. It works great.

Ever heard of a not so new tool called IDE? You are welcome.

my whole computer is my integrated development environment =)

Applications are open for YC Summer 2018

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact