
GDB Debugging Full Example: Ncurses - okket
http://www.brendangregg.com/blog/2016-08-09/gdb-example-ncurses.html
======
tomjakubowski
One of the coolest days I've had at work was when we hit a nasty deadlock deep
in the bowels of gstreamer on one of our shared "immersion room" computers. We
pinged our resident gstreamer expert and I convinced him to run his gdb
session (from a core dump, over ssh, halfway around the world) inside tmux so
we could watch him as he debugged.

There's nothing quite like watching someone who's an expert both in gdb and
the problem domain as they do their thing.

~~~
fapjacks
"Immersion room" computers? What do you do for a living?

~~~
tomjakubowski
We develop interactive 3D applications at "room scale".
[http://www.oblong.com/solutions/](http://www.oblong.com/solutions/)

The top photo is of our immersion room staging environment. The versions
deployed to production (at a handful of sites worldwide) have a little more
polish :-)

~~~
fapjacks
Thanks, and sorry, this is totally off-topic... I am reading this as saying
that there is some kind of data (or multiple kinds of data) which are worth
viewing in multiple dimensions in giant holodeck-like rooms. I'm assuming
remote conferencing is one of those, but I am interested in hearing more about
the other kinds of data. Do you have some links to something a little more
concrete than what's on the site you linked?

------
iopuy
I just want to express how nice it is to see someone use vanilla tools on
Linux to solve a problem. Whenever I see a thread on using gdb or jdb,
invariably it is full of suggestions on the use of a half baked debugging
wrapper that hasn't been maintained in 2+ years under the guise of it somehow
being better because it has colors.

------
kaushiks
You can also launch the TUI after-the-fact using the key sequence 'C-x o'. (It
is no accident that it is the key sequence bound to other-window by default in
Emacs).

------
qwertyuiop924
It's so surreal to see Brendan Gregg discussing Linux tools.

It's like watching Iron Man wielding Thor's hammer.

Which is not to compare Brendan to Iron Man, but it's as weird as that would
be.

~~~
brendangregg
Thanks, but I should stress that my example here didn't do much that was
really advanced. There was a little ftrace and BPF, gdb reverse stepping and
the TUI, but the other gdb commands are routine. I guess what's useful is
explaining everything, including caveats that you can hit even with the basic
stuff.

~~~
qwertyuiop924
No, I'm not complaining, it's just odd seeing you primarily work with solaris
for so many years, and now talking about linux toolsets instead. I mean, I
still remember you dissmissing SystemTap, although I believe you later said it
had improved...

Speaking of which, as I understand it, dbx is to gdb as adb is to mdb. So
other than syntax, what IS difference between the adb and dbx families, as
you've clearly used both?

------
FreeFull
As far as reverse debugging goes, you really should look into [http://rr-
project.org/](http://rr-project.org/)

~~~
brendangregg
I've been meaning to. I took a quick look now:

    
    
      # rr record date
      rr: Saving the execution of `date' to trace directory `/root/.local/share/rr/date-30'.
      [FATAL /build/rr-jR8ti5/rr-4.1.0/src/PerfCounters.cc:195:start_counter() errno: 2 'No such file or directory'] 
       -> Unable to open performance counter with 'perf_event_open'; are perf events enabled? Try 'perf record'.
      Aborted
    

oh-oh...

    
    
      # gdb `which rr`
      [...]
      (gdb) r record date
      Starting program: /usr/bin/rr record date
      [...]
      [FATAL /build/rr-jR8ti5/rr-4.1.0/src/PerfCounters.cc:195:start_counter() errno: 2 'No such file or directory'] 
       -> Unable to open performance counter with 'perf_event_open'; are perf events enabled? Try 'perf record'.
      Thread 1 "rr" received signal SIGABRT, Aborted.
      0x00007ffff6d6b418 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
      54	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
      (gdb) bt
      #0  0x00007ffff6d6b418 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
      #1  0x00007ffff6d6d01a in __GI_abort () at abort.c:89
      #2  0x00000000005a9133 in FatalOstream::~FatalOstream() ()
      #3  0x00000000006303fc in ?? ()
      #4  0x0000000000630679 in PerfCounters::reset(long) ()
      #5  0x00000000006cbd8f in Task::resume_execution(ResumeRequest, WaitRequest, TicksRequest, int) ()
      #6  0x0000000000638be9 in RecordSession::task_continue(Task*, RecordSession::StepState const&) ()
      #7  0x000000000063d70e in RecordSession::record_step() ()
      #8  0x00000000006354b6 in ?? ()
      #9  0x00000000006356f2 in RecordCommand::run(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) ()
      #10 0x000000000061f44d in main ()
    

I did a bit more gdb debugging (one debugger debugging another, although it's
a bit hard as rr is stripped and doesn't have a debuginfo package!), and it
looks like it's trying to use PMCs.

No PMCs here: EC2 Xen guest. So rr won't work, for me or EC2's 1+ million
other customers. Perhaps rr can switch to software events?

------
_cairn
There's a typo under heading 7.

...And thanks for the awesome write-up, makes something intimidating for a
beginner like myself (gdb) seem somehow less so. Feeling inspired.

~~~
brendangregg
thanks, fixed!

I was trying to make it not intimidating by just writing what I knew, and
_not_ researching all the odds and ends, instead, just saying "I don't know"
etc as appropriate. Plus including my mistakes.

~~~
_cairn
I'd like to add, as someone who is a novice programmer and trying to get more
and more immersed in this world - it's posts like these that give you a small
peek into _thinking /being_ a programmer as opposed to _doing_ programming as
a job/profession/etc. Part of what makes programming and computers scary to
beginners, speaking from my experience, is the notion that there is an
infinite amount of complexity hiding just behind your IDE (things far beyond
your grasp/ability to comprehend - i.e. You'll never know _really_ how your
program goes from the pretty syntax highlighted code in the IntelliJ window to
x y or z app, or worse as in this case, when something goes wrong god forbid
something with a sinister sounding name like 'segmentation fault'). Breaking
that barrier down through the use of seemingly simple yet extremely powerful
tools (Emacs, gdb, bash, etc) really helps, at least for me, feel less raw
intimidation and self-loathing and more and more wonderment and inspiration to
continue learning and hacking away.

------
pixel_fcker
Thanks for this. As someone who hasn't got much further than walking a stack
trace and printing variables in gdb it's incredibly helpful.

------
busterarm
Did everybody just discover the TUI in 2016 or something?

Not being flippant, just wondering because there've been a ton of links about
it lately.

~~~
brendangregg
Probably, thanks to Greg Law's talk which featured here on HN, which I linked
to in the post.

But this is often the case with Linux kernel and GNU tool features: they don't
appear to have any marketing or sales professionals telling everyone about new
features (unlike commercial offerings), which leaves us with many "hidden
gems" (as people would say) in Linux + GNU. ftrace is another example.

