
Debuggers Suck - samth
https://robert.ocallahan.org/2019/11/your-debugger-sucks.html
======
sfink
I will corroborate one main point of the article: debugging with rr is so much
better than a traditional debugger that I will only buy Intel hardware, even
with all of the security flaws and corresponding performance hits, even though
recent AMD CPUs would halve my half-hour compile time.

It really is that much better. Once you start using it, it's really hard to go
back to a situation where you don't have it. It's like driving a car that can
only turn left -- sure, if you need to go right you can always 270 degrees to
the left (by rerunning over and over again to get closer to what you want to
examine, assuming your problem is reproducible enough), but it feels
burdensome and ridiculous.

If AMD fixed their performance counters so that rr could work, I would switch
immediately.

I am also a fan of Pernosco. It is especially useful when hooked up to a
continuous integration system that can hand you a ready-to-go recording of all
failures. You can get assigned a bug with a recording to go along with it.
Even if it doesn't look important or tractable to look into the ordinary way,
having a recording makes it easy enough to take a glance and check it out that
you'll actually do it. At shops that do a lot of system-level programming (as
opposed to scripting), the productivity boost ought to be worth a serious
amount to the bean counters.

~~~
de_watcher
The wild thing is that scripting languages don't have that kind of a debugger.

~~~
yvdriess
Depends. Image-based runtime languages such as Smalltalk and various Lisps
will, crudely put, pause at the site of the error and give you options to fix
it and continue. That already covers 99% of the OP's issues with gdb.

edit: Implied is that image-based runtimes by definition produce reproducible
snapshots of the error, with the full power of the Lisp or Smalltalk
introspection tools. Edit-continue is the icing.

~~~
roca
OP here. Edit-and-continue addresses _none_ of my issues with gdb. Read the
paragraph "If you use a traditional interactive debugger..." again?

~~~
yvdriess
I think you are missing the implications of working with image-based language
runtimes.

The CI or even the client can attaches the live image of the error, the
programmer opens up the image when he opens the ticket and has the full
venerable introspection tools of CL or Smalltalk. This directly addresses the
reproducability issue you raised in said paragraph. There are indeed occasions
where you need a proper rr-like trace, but what fraction of bugs fall into
that category.

For illustration purposes: Grammarly has an example of fixing an intermittent
network bug on a production server under load on the much-HN-linked blog post:
[https://tech.grammarly.com/blog/running-lisp-in-
production](https://tech.grammarly.com/blog/running-lisp-in-production)

~~~
db48x
I agree that debugging in Common Lisp is much more powerful than debugging in
GDB. For one thing, when you get a condition and your repl enters the
debugger, you immediately learn what condition handler you should add to your
code so that it can automatically handle this case gracefully in the future.
And you have the full compiler available to you in that repl, so you can
redefine functions, change variables, and so on. This lets you recover from
the error and leave the program running and in a better state than it was
before you started. This is actually a super power that you just don't have
when you're debugging a program written in C. (Technically I suppose some IDEs
have an edit-and-continue feature, but I've never heard of anyone using it to
save a program running in production.)

On the other hand, rr gives you a different set of features. You can rewind
the full state of the program to any previous point in time. I've used rr a
lot, and this capability is extremely useful. It makes debugging heap
corruption, buffer overflows, use-after-free, and many other complicated bugs
much, much easier. You just set a watchpoint on the data that was overwritten,
then run the program in reverse (using the reverse-continue command) until it
gets back to the point where the write happened. It really is like having a
super power, but it's a different kind of super power than the Common Lisp
repl.

Pernosco is another level above that. Instead of presenting you with the state
of the program at a single moment in time and letting you run it backwards and
forwards, it lets you query the full state of the program. Where before you
would set a watchpoint and run backwards until you got to the previous write,
with Pernosco you click on the value and it presents you with a list of _all_
writes to _all_ locations in the chain that leads to that value. For example,
you can see that the value was written to the memory location from a register,
and that before that it was read from some other memory location into the
register, etc. It's really very cool.

There's no reason why a Common Lisp or Smalltalk runtime couldn't have these
features in addition to a proper repl.

~~~
greggman2
I have dumb question

> You can rewind the full state of the program to any previous point in time

Where does all this saved state go? I'm running a game that runs at 60fps,
every frame 10s of megs of data change. It won't take more than a few second
to run out of memory to save all the state. Further, if the game doesn't run
at 60fps (or 90fps on VR) it's not usable, if saving this state slows the game
down to 20fps or 15fps I can't tell if my code is working since the entire
goal is to run at speed and check it all feels good.

Is there some technique these kinds of tools use to make it possible to do
keep all changed state in this type of app or do they only work for some other
type of app?

~~~
garaetjjte
By capturing all external inputs to application, it can replay it many times
and results will be always identical.

[https://arxiv.org/pdf/1705.05937](https://arxiv.org/pdf/1705.05937)

------
jpochtar
As someone who's also worked on a debugger, I'm sad to say many programmers
aren't working on problems that needed complex debugging. I couldn't
understand why friends weren't interested in what I was working on, until I
realized they were building CRUD apps all day.

There's little control flow to speak of in there, so if they had a problem,
they could just dump locals() to get a good view of everything that happened.

When I went back to doing compilers work, I missed my debugger dearly. I also
realized why I needed a debugger and my friends didn't: debuggers like this
are particularly useful for work with compiler-like characteristics, of which
there is unfortunately not enough.

~~~
zer00eyz
> I couldn't understand why friends weren't interested in what I was working
> on, until I realized they were building CRUD apps all day.

As someone who gets paid to clean up the cluster fuck that can be created by
people building "crud" apps let me tell you better debugging tools would be a
godsend.

In the old days with more monolithic apps you at least had one pile everyone
shit in, and it was possible to get a handle on how it "functioned" and what
its quirks were. Between micro-services and docker it isn't one big shit pile
its a bunch of small ones, everywhere. And because of docker on occasion I now
get to dig down to the underlying OS, because someone who had no right
touching a production server, ever, built and tweaked a container and it
"runs" so it went out. Did you know that you can run diff over ssh to compare
and contrast the code on two containers? I can think of two rather heathy pay
days in the last year where some Jr. on site engineer is the one who found the
problem by going bottom up while I went top down.

The article states that "developers suck" and on a bad day I would say that
the author is giving them too much credit. On a good day I realize that the
issue is much deeper and that we hire technological magpies who want to chase
shiny new objects (the javascript world is probably the worst at this). On a
really bad day I realize that our entire industry is "wrong headed" starting
from hiring. We test people on "can they write code" but no one ever asks the
people walking through the door to read 50 pages of it. To find the one
misplaced semicolon just to see if they picked up on any of the logic traps
that were littered all over the other 49 pages.

So to you sir (or madam), I raise a glass for at least trying to build tools
for me, the guy who fixes the "gnarly bugs" in crud apps. I wish you would go
back and finish your work, the 6 of us who really appreciate you would happily
buy you a beer.

~~~
james_s_tayler
The time, patience and level of enthusiasm required to learn the craft to the
level you can develop a piece of software that is well-designed, performant,
easy to reason about and an overall joy to work with is about 10 to 100x what
the average engineer is willing to put in. It's a hard career and industry for
this reason. We all know the systems we work on are full of subpar choices,
design flaws. We all know the delta between what it is an what it could be.
But almost nobody can reconcile that gap on their own. And multiply that by
entire teams. It's orders of magnitude less likely an entire team is that
skilled and enthusiastic.

I have three philosophical perspectives that help me better frame the software
industry.

Number one: People aren't stupid. Rather, things are hard.

Number two: Writing software is like doing a trillion piece jigsaw puzzle.

Number 3: Software Engineers are like Surgeons, except the twelve years of
training is received on the job.

~~~
LandR
Number 3: MOst SOftware Engineers are bad at their job, they aren't anything
close to Surgeons.

This I think is the reason most software nowadays is garbage, there is a lot
of low skilled people creating it. I blame bootcamps and dumbed down CS
courses for this. It's not necessarily the developers fault, they can't
improve on the things they don't know they are bad at.

~~~
nradov
Computer Science curriculum was never intended to produce competent software
developers. You're complaining about the wrong problem.

------
KenoFischer
Another testimonial from me. I basically refuse to do any debugging on systems
that rr doesn't run on unless there's a very good reason. I use rr extensively
whenever there's a bug in Julia and I would singlehandedly credit it with a
measurable increase in system stability due to it catching bugs we'd never
have gotten otherwise. roc is one of the few people with both the vision and
persistence to really improve the debugging experience.

~~~
mrspatula
What's the best way to use rr to debug Julia code atm? Or are you talking
about debugging the actual language code in C?

~~~
KenoFischer
Yes, the latter (mostly for various weird corner cases). Back in the
Gallium.jl days we could actually debug the recorded julia code, but we
haven't regained that capability yet.

------
pjmlp
Maybe I missed something, but what really sucks is making "how to use a
debugger" demos, because most developers don't even know what their debugger
can do for them, let alone use it properly.

Visual Studio team started doing VS productivity talks, because they noticed
that many feature requests were about features that VS already supports for
ages.

~~~
int_19h
It can be worse. I work on Python debugger in Visual Studio and VS Code, and
it's still unbelievable just how often, when I describe what I do to people
who write production code in Python, I have to explain what e.g. conditional
breakpoints even are, and then they say things like, "Wait, there is a tool
that can do _that_? That could have saved me a few hours the other day!". And
this all has been around for many years now...

There's some progress - these days it's mostly confined to the data science
crowd, and web devs have mostly caught up, due to the rapidly increasing
popularity of PyCharm and VS Code over the past few years.

In general, it feels like there's some weird gulf when it comes to tooling. On
one side, you have the hardcore C gurus (often doing embedded), who do heavy
scripting in gdb to solve very complex problems. On the other side, you have
IDEs that grew out of 1990s RAD - Eclipse, Visual Studio etc - in ecosystems
where the language and the tooling are usually developed and packaged together
(e.g. whenever a C# compiler ships a new language feature in stable, there's
also a new stable version of VS that supports it in the editor, the debugger
etc - and UX for those is explicitly considered when designing features). And
in the middle, you have all the high-level "scripting" languages, which have
both low-level debuggers and powerful high-level IDEs - but their respective
dev cultures largely ignore both for historical reasons.

~~~
mbeex
There is more to it. In the past, I was a all-day VS User (C++ most of the
time, but even Python. Today I prefer Wing IDE for this (and sometimes VS
Code)). While conditional breakpoints can be very useful, they often are not
(especially because they slow down the code - hitting a variable value after
20000 iterations in a heavy-loaded loop as an example). It follows, for most
developers syntax doesnt sink in. At least in the past, documentation was too
far away. These things have to be literally at your finger tips. Don't know
about the situation nowadays, though.

~~~
pjmlp
VS shows you a sample condition and you can follow up for the related
documentation.

------
bsder
As much as I love rants like these, the problem with debuggers is
_discoverability_.

I have lost track of the number of times that I kill myself debugging
something with gdb only to later explain it to some group and somebody says:
"LOL, n00b. Why didn't you just use <magic gdb command>?"

Why? Because I didn't know about the magic gdb command and there was no
documentation anywhere that would have pointed me to the fact as to that
command being exactly what I was busting my head over.

Debuggers are like build systems. You don't diddle with them every day and
when you are something is wrong and you're on a deadline.

If people want programmers to appreciate better debugging, we need some good
videos of people debugging real problems in real time with debuggers.

~~~
roca
OP here. I get it. I led rr development and even I sometimes find gdb features
I never knew about!

We explicitly tried hard to make Pernosco discoverable, partly by creating
simple but powerful and composable features, and partly using traditional GUI
techniques. We definitely haven't fully succeeded, but I think we moved the
needle. If you dig through
[https://pernos.co/about/](https://pernos.co/about/) you can see what we
tried, and if you try the demo
[https://pernos.co/debug/e9UFXkGq__8KQB-N6n59lA/index.html](https://pernos.co/debug/e9UFXkGq__8KQB-N6n59lA/index.html)
you can see how well we succeeded.

~~~
ahaferburg
FYI none of the images on [https://pernos.co/about](https://pernos.co/about)
are working for me.

> _static.pernos.co has a security policy called HTTP Strict Transport
> Security (HSTS), which means that Firefox can only connect to it securely.
> You can’t add an exception to visit this site._

> _The issue is most likely with the website, and there is nothing you can do
> to resolve it._

~~~
roca
Are you using Firefox Nightly? It had a bug not long ago which was incorrectly
blocking those images.

------
mantap
It would be amazing if someone could design a programming language around
debugging. There are many languages that are designed around preemptively
catching bugs using a type system, and I do use static type systems, but this
in no way oblivates the need for a debugger.

The way I want to write code is directly as it is executing. I think the
distinction between writing code in an editor and debugging is arbitrary - I
want to edit programs as they are running and see the data as it flows through
beside the code. I want to be able to browse the object graph as if it were a
filesystem like I could in Objective-C (I think Smalltalk invented this?).

This does require a different code architecture to be effective: more pure
functions and less tangled state, but those are usually considered to be good
things. Improving debugging might have a positive effect on code quality.

~~~
dwohnitmok
Clojure is both a good and bad example of this. Live hot-reloading means that
you really are editing programs as they are running. This is especially
visceral if you're working on something with a GUI component as you can see
the GUI change before your eyes as you work on it. And if you're really
adventurous you can open up a REPL connection to a running program in
production and edit your production service as it's running (don't worry you
can disable this so that random folks can't start modifying your production
code)!

On the other hand, running a debugger in Clojure is a bit of a mess. Generally
I just don't and end up creating "watchdog" atoms that hold bits of state that
I think are relevant to the situation at hand and evaluate those with a live
REPL connection. See e.g. [https://eli.thegreenplace.net/2017/notes-on-
debugging-clojur...](https://eli.thegreenplace.net/2017/notes-on-debugging-
clojure-code/). Although the article ends on an optimistic tone, I've found
using a debugger with Java or Scala more rewarding than using one with
Clojure.

~~~
michaelmrose
Have you seen sayid and the cider debugger?

------
thayne
> In particular, developer culture is that developers don't pay for tools,
> especially not debuggers

I think there are a couple of reasons for this. One is that for many if not
most developer tools, there are free options that are at least competitive
with paid tools. Another is that developers can, at least in theory, read
source code to better understand a tool they use, or even improve it, and
therefore want access to the source code. Which means open source tools have
an advantage over proprietary tools in addition to price.

~~~
DC-3
This line galled me too as it showed a serious misunderstanding of the
principles of libre software. There's a reason we say Free as in Free Speech,
not Free Beer! Why would I ever want to make my workflow dependant on a third
party blob whose internals are obscure from me?

~~~
leoedin
The price does also play a role. Learning tools and frameworks takes a lot of
time, and if it's not a free tool then if you change jobs or even departments
you might find yourself unable to get a license. If you want to work on stuff
at home, you need to pay. If you want to run a test server you need a license.
Some tools are better than others in that regard, but it's basically why I
would always choose learning a slightly worse free option than a paid option.

~~~
thayne
Oh i wasn't saying that cost doesn't matter, just that it isn't the only
factor.

------
garaetjjte
>my messages about Pernosco, which I think is much more interesting and
important, have relatively little exposure

It is not very surprising given that it is a SaaS which doesn't even list
pricing on its page.

>I want everyone to know about Pernosco, not just to attract the customers we
need for sustainable debugging investment, but so that developers everywhere
wake up to the awful state of debugging and rise up to demand an end to it.

This is rather conflicting statement as Pernosco business model seems to be
solely targeted at bigcorps paying big money, and not developers promoting
Pernosco through using it in own hobby projects. (I don't know if this is
actually true, but email-us-to-just-get-a-quote is huge discouragement)

~~~
mepiethree
It is also very difficult to reach a usage example or a workflow on their
site. There is no demo. I agree that debuggers suck, but I don't see any
clear/obvious evidence that Pernosco doesn't suck. I really just have to take
their word for it.

I don't want to pay an undisclosed amount of money for the _opportunity_ to
spend an unknown amount of time to get started with a new tool that may very
well suck.

~~~
roca
There's a Youtube video right on the landing page:
[https://pernos.co](https://pernos.co) That video links to where you can test
that session out for yourself:
[https://pernos.co/debug/e9UFXkGq__8KQB-N6n59lA/index.html](https://pernos.co/debug/e9UFXkGq__8KQB-N6n59lA/index.html)

------
jnwatson
I used to sell debuggers a decade ago. We even had reversible debugging. The
amount of personal development productivity I had was unmatched; I’ve never
gotten back to that point.

It is true, nobody wants to pay for engineering tools. I wish the author luck.

~~~
m463
The market is so small, and there appears to be lots of premature optimization
of the checkbook.

I remember many years ago trying out purify, and it revolutionized my
debugging. The problem seemed to be that the program I wrote could be debugged
by everyone - if you just got everyone a per-user license.

It would have been hard enough to justify a license for me (a person at the
bottom with no power), it was impossible to justify a license for everybody.

~~~
TeMPOraL
> _It would have been hard enough to justify a license for me (a person at the
> bottom with no power), it was impossible to justify a license for
> everybody._

"Floating" licenses are better here. In my experience, companies are reluctant
to buy licenses for an entire team for something that's used only
occasionally. It's easier to make them buy a few floating licenses to be
shared by the team.

~~~
m463
I am uncertain if floating licenses were available at the time. Even so, when
floating licenses became popular, there was also the "license in use" denial
problem.

Unfortunately there's another way to do this nowadays. Offer convenient tools
with analytics, etc... (and coincidentally spy on everyone)

------
BubRoss
I agree but this whole thing just says 'debugging sucks, my commercial tool is
great, I won't explain a single detail about it, follow this link' (it is a
wrapper around rr). He's preaching to the choir and I still think this post is
sleazy.

Also replay debugging doesn't have to be complicated.

This enables replaying sections of a program with hot reloading by requiring
that the data structures are serialized. It also catches low level exceptions
so that replay and hot reloading won't crash the running program.

[https://github.com/LiveAsynchronousVisualizedArchitecture/la...](https://github.com/LiveAsynchronousVisualizedArchitecture/lava)

~~~
heycam
There's a great series of posts going into more detail about Pernosco's design
and advantages that is linked to in the final paragraph:
[https://pernos.co/about/](https://pernos.co/about/)

------
davidhyde
The Visual Studio .net (c# in particular) debugging experience has always been
excellent in my opinion. One great plus for a managed language. I guess that
the closer you get to the metal the more difficult it is for a debugger to do
its magical things and hence the poor experience for c++ devs.

~~~
roca
Apart from Intellitrace, AFAICT it's a great implementation of the traditional
stop-and-inspect debugger feature set, but that's all. Intellitrace adds
something, but not full access to all program states like rr (or Pernosco). So
I think your expectations could be higher :-).

~~~
GordonS
> it's a great implementation of the traditional stop-and-inspect debugger
> feature set, but that's all

It provides more than that - the "Edit and Continue" feature is invaluable,
allowing you to pause execution at a breakpoint and change the code, such that
the new code runs when execution resumes.

As an aside, Rider also has this feature, and I find it to be much faster and
more reliable than VS.

Visual Studio also recently got "Time Travel Debugging"[0], which lets you
record execution then replay in the debugger. I haven't actually tried this
yet.

[0] [https://devblogs.microsoft.com/visualstudio/introducing-
time...](https://devblogs.microsoft.com/visualstudio/introducing-time-travel-
debugging-for-visual-studio-enterprise-2019/)

~~~
roca
Ah yes, TTD is a real game-changer. It compares fairly well with rr, though
the recording overhead is often a lot higher. Lacks a lot of the Pernosco
feature set though.

I think that not many people have used TTD yet so when people talk about how
great the VS debugger is, they're not really talking about TTD.

~~~
GordonS
I think Visual Studio's TTD is still only available with Azure web apps or
VMs, which certainly limits the audience. Hopefully it will be made more
widely available soon.

------
modeless
I dearly wish that I could use rr and Pernosco, but I do graphics work and
GPUs don't work with fancy debuggers. The state of GPU debugging is even more
dire than CPU debugging.

~~~
uonyx
A debugger like this (sort of) already exists for GPU work (but for Metal).

[https://developer.apple.com/documentation/metal/shader_autho...](https://developer.apple.com/documentation/metal/shader_authoring/developing_and_debugging_metal_shaders)

[https://developer.apple.com/videos/play/wwdc2018/608/](https://developer.apple.com/videos/play/wwdc2018/608/)

------
comex
I‘d be willing to pay for a better debugger.

But if the debugger doesn’t come with source code that I can inspect and
modify, that’s a big negative that will have to compete with whatever
advantages the debugger has. (It’s not just a hypothetical use case. I’ve
grepped through the source code of GDB and LLDB on several occasions to figure
out why they weren’t working properly.)

And if the debugger is SaaS, meaning it will _almost certainly_ disappear in a
few years, after I’ve gotten used to it… it would have to be some super magic
unicorn stuff for me to consider touching it, even if it were free.

Pernosco is both closed-source and SaaS. Pass.

~~~
sfink
Try out rr first. It has at least half the magical unicorn sparkles you're
looking for, is open source, and runs locally. It's not hard to modify. (I've
done it, though not for any core functionality changes.)

Then you'll be in a far better position to evaluate the additional benefits
Pernosco might provide you. I think your reasons for being leery of it are
valid, but there are very legitimate scenarios where it makes sense IMO.
(Recording CI jobs automatically, collaborative debugging, situations where
the critical dataflow is hard to follow...) And they integrate, too - you can
upload rr recordings to Pernosco when you need the extra functionality.

I guess it also depends on the pricing model, which I know nothing about.

------
gHosts
I once thought I could do better than gdb....

So I dug down into the bowels to see why it's so lame......

Oh my.

The hideous complexity imposed by (multiple) CPU designs, OS interfaces, the
vast mismatch between the code as written and the code as compiled. The
complexity of the DWARF standard....

Sigh! Sorry gdb dev's... I'm sorry I ever thought bad of you guys...

You guys are doing way better than I'd ever do.

------
xwdv
Tragically, one of my regrets in life is never really learning proper
debugging. It is embarrassing that even as a senior engineer in the industry,
the vast majority of my debugging is basically a bunch of console logs, and it
gives me a serious case of imposter syndrome when I see others using all sorts
of fancy debugging tools.

If there is a silver lining, it must be that my lack of debugger skills can
only mean I spend a lot of time writing safe, bug free code. Or more likely,
no code at all.

~~~
sfink
Fixing bugs without a debugger is a great skill. If you're good at it, you'll
be able to figure out many problems far faster than people who depend on
debuggers, and you'll be more likely to get to root causes and change things
to be more robust at an architectural level.

Fixing bugs with a debugger is a great skill. If you're good at it, you'll be
able to figure out many problems far faster than people who don't use
debuggers. You can skip a lot of deep thinking and go straight to the site of
a problem. You can explore the actual execution paths in ways you'd never
figure out from all the layers of templates and overloads. You can debug other
people's code almost as easily as your own. You won't get caught by some
subtly wrong assumption you're making when thinking through how things are
supposed to work.

Debuggers are dangerous. They're insanely useful, but once you start depending
on them, your ability to fix things without a debugger will atrophy. If I had
the self-discipline, I'd ban myself from using debuggers for a month out of
every year.

------
jdblair
There's no mention here of inspecting core files, something that very much
allows you to inspect the state of a program when the bug occurred, but after
the fact. Is this because inspecting core files also sucks, since it's a
snapshot of a point in time, and it's no use if your stack is corrupted? Or is
it because this is a lost art?

In my embedded work I avoided using debuggers because setting up remote
debugging can be a PITA. I figured out a long time ago that building remote
debug setup into my workflow will save oodles of time in the end. Still, I
find myself using plain old logging more often than not - the bug may not be
in C or C++, it may be on the JavaScript run in the interpreter embedded in
that C++ application.

~~~
gmueckl
Core dumps are great, especially for the occasional odd bug that occurs only
in long running processes. However, the ridiculous systemd default of
attempting and failing to capture the kernel core dump into the binary journal
completely ruins this. Only small, uninteresting core dumps are ever captured;
large ones are silently discarded because the journal is not allowed to grow
that big.

~~~
sfink
Yeah, when I'm crashing and want to keep cores, I'll temporarily modify
/proc/sys/kernel/core_pattern to prevent systemd from eating them. I sometimes
want them suppressed, sometimes want them written out, and never want them to
go to systemd.

~~~
Pahr3yah
One issue is that that sysctl is not a namespaced one.

------
wheybags
I don't care much about fancy features. I just want a simple debugger GUI in
my IDE that works consistently, and never crashes. Until I have that, I feel
like pursuing even more functionality is pointless. (Context: c++ developer on
Linux. The only debugging experience I've ever had that approaches what I
describe was in c# on visual studio)

------
sansnomme
A lot of debugging UX is editor integration. For top end IDEs, these are
available out of the box. Set breakpoint by clicking left margin of any kind
of code, another click to run trigger debugging. Similar story for VS Code
though a bit more configuration is required. It's only an issue when it comes
to lightweights like Sublime Text/Emacs/Vim. The philosophy of things being
"loosely coupled" means that you have to be prepared to spend an hour browsing
man pages just to figure out how to create a breakpoint. Edit and continue
isn't just a fancy gimmick used in Lisp; it's a first class feature in Visual
Studio (iirc it originated in the original Visual Basic, got removed during
the transition to .NET before being added back again to Visual Studio, someone
from MSFT should confirm this) Python is just "rediscovering" how to do this.
JavaScript (in Chrome at least) have partial support, mostly because they were
forced to do livereloading so frontend development isn't too unpleasant. Rust
and Go and all the new age languages never seemed to have heard of it despite
proclaiming their commitment to good developer debugging process (Elm _might_
have some support though their blog releases seemed more interested in time
traveling functional data structures rather than benefits to developer
experience, and no, you do not need fancy stateless functions to add Edit and
Continue though I suppose they do make it easier).

~~~
vardump
How about being able to debug multiple threads and processes? That would be
actually useful.

> Rust and Go and all the new age languages never seemed to have heard of it
> despite proclaiming their commitment

Go rebuild / full rebuild times are fast enough. I don't think edit and
continue would be that amazing with 1000 concurrent goroutines.

Rust will hopefully have edit and continue one day. Hopefully compiler will
get faster as well... That said, I much prefer Rust compiler catching a
significant portion of issues at compilation phase, thus not requiring
debugging in the first place.

~~~
int_19h
I don't think I've ever seen an IDE for a multithreaded language that couldn't
debug multiple threads. Most can also debug multiple processes in a single
session, and some can automatically attach to child processes.

~~~
vardump
I was thinking more about everything timeouting (or otherwise losing relevant
state) around debugged thread / process.

~~~
sanxiyn
rr completely solves this problem.

~~~
vardump
I both use Windows and deal with software using shared memory. Neither of
which rr can deal with.

While rr seems to be very nice, there's still room for improvement. But I
guess this kind of tool might be impossible on current hardware, at least
without emulation.

------
pizlonator
This is such a great rant.

One bit I’d add is the bad attitude towards debugging in optimizing compilers.
It’s not obvious to me that optimizing for perf at the cost of debugging
fidelity is a good trade off but here we are. Like, if you could save enough
time debugging with better tools then you’d have more time to optimize. And
maybe if compiler devs confronted full-speed debugging head on then we’d get
to perf that is good enough for everyone and we’d all be more productive.

Or maybe it all sucks and all we can do is bitch about it.

~~~
roca
Glad you enjoyed it.

For clang users things would get a lot better if someone just fixed a lot of
clang bugs.

Omniscient debugging creates new opportunities to overcome the impact of
optimizations. E.g. are you trying to display the value of a variable after
its live range ended? No problem, look back in time to when it was live.
However, in many cases we need extensions to debuginfo to take advantage of
that; e.g. DWARF can't express the difference between "variable outside live
range, but has not been modified" and "we don't know where it is".

If we had a debug build and an opt build of the same program then there are
some mad-science approaches that would be fun to try.

So I actually think the perf/debuggability tradeoff is not fundamental and
with the right resources we could mostly make it go away.

~~~
pizlonator
JSC tries to go into the tricks you describe for the purpose of optimizing OSR
exit. It’s really hard. There is a register pressure overhead from situations
where the program thought a thing was live but really at that program point
some of the necessary inputs to the value you’re trying to recover (and the
value itself) would have been trashed by regalloc if it wasn’t for the need to
be able to rematerialize that value. That translates into register pressure
and you can feel it even on forgiving platforms like arm64.

I believe that this overhead in conditions where the codegen is otherwise
behaving well is around 5%. So not terrible, but that’s my call of the price
relative to -O3 or whatever.

~~~
roca
Perhaps I wasn't clear. None of things I mentioned would entail changing the
generated code in any way.

In the comment about omniscience, here's an example of what I'm proposing: \--
Debuginfo says variable 'x' is in register R1 in the PC range 0-10 \-- At 10,
R1 is set to something unrelated \-- Debuginfo says in the PC range 10-20, 'x'
is equal to its last live value (i.e. its value at PC=10) Pernosco would
evaluate the value of 'x' at PC=15 by actually reconstructing the value in R1
at PC=10, using the omniscient database we already have.

Of course DWARF can't currently express this, because normal debuggers can't
make use of it. Even in rr it would be quite expensive to compute.

If you're familiar with DW_OP_entry_value, it's a bit like that. That's a way
for DWARF to say "the value of this variable (e.g. a parameter) is equal to
the value of this subexpression (usually a register) on entry to the
function", which regular debuggers can often obtain using stack unwinding.
Pernosco doesn't do the unwinding, instead it (logically) warps back in time
to the point where the function was called and evaluates the subexpression in
that context.

~~~
pizlonator
Gotcha. I get what you’re saying now: you just replay and it’s all good.

------
Crinus
> As far as I know, the only way to fix this situation is by building tools so
> much better than the free tools that the absurdity of refusing to pay for
> them is overwhelming, and expectations shift.

This of course will only work for a little while until someone who has more
time than money (or is a big company that wants to commoditize the tool) will
build a command-line version of it on Linux. Weird UX and the need for a
spaghetti of shell scripts to integrate with vim/emacs/vscode/sublime/ed will
soon follow with an Eclipse addon that nobody will use a bit later. After a
macOS port, assuming these are still a thing, Apple may create a nice looking
UI and integrate it with Xcode or someone like Panic may create a good front
end and sell it for ~$99.

Most people will keep using Windows, think Visual Studio has the best debugger
and everyone will be happy.

Those that learn about the "overwhelmingly better approach" will consider the
free one the best one and its spaghetti of shell scripts approach the obvious
best approach, because they wont have any real deep experience with the paid
tools (either the original one or the macOS shiny wrapper - which will be
considered as unnecessary by most anyway) to properly judge.

~~~
roca
Yeah, nah. I sympathize with your point of view, but implementing Pernosco
requires some serious science. If you try building a database of all memory
and register changes in a naive way, it isn't going to work at all for
nontrivial programs. Of course it could be cloned, but you would need a very
good team and a lot of work.

Also either you start with rr as a base, in which case you need people with
deep rr knowledge, and I know their names, or you build or buy your own record
and replay framework --- more time and money.

~~~
souprock
It's no longer likely at all, but some years ago my workplace nearly open
sourced a serious tool. That would have been big news, with the sudden
availability of source for a tool that does rr-like stuff.

Some other company could do it. Remember that we've gotten Navigator (now
Firefox), Blender, OpenSolaris, OpenOffice, and the .net stuff. Governments
can surprise us as well; the USA did that with Ghidra.

I'm not expecting it anytime soon, but surprises happen.

~~~
roca
That's a possibility, but I don't know of any closed-source tool that does
what Pernosco does. Tetrane is the closest I know of, but they aren't as
scalable as Pernosco (they don't need to be).

Certainly possible that someone has a secret tool that's as good or better.
However, I expect the secret tools in this space are all focused on so-called
'security research', i.e. more like Tetrane and targeted at adversaries of the
software under test, rather than its developers, which leads to a somewhat
different feature set.

------
alkonaut
I think some times the conversations about debuggers is one where developers
talk past eachother because a developer who uses a high level language
debugger in a fancy IDE to debug only high level code, vs. one that uses a
native language debugger using a command line tool, will have wildly differnet
experiences and expectations of what the tool can do, and especially about how
discoverable and simple those features are.

I belong in the first camp (Debugging to me means "running", It's not a tool I
dig out to actually de-BUG something, it's just the tool I use to run my code
every time).

~~~
clarry
They also talk past each other because the value of debuggers vastly depends
on what you're debugging.

It's relatively easy to get value out of a debugger in a straightforward
synchronous userland process.

It's much harder in a complex embedded application full of threads, callbacks,
asynchronous execution, and a target where you may need to set up something
like remote debugging and potentially tweak the whole system config to make it
possible in the first place. This is also more likely to be the kind of system
that doesn't run from your fancy IDE..

------
devin
I do almost all of my work in Clojure and have for nearly 7 years. I don’t
actually miss a traditional debugger much. I’ve occasionally tried one out,
but an interactive REPL and immutable data means printlns with the occasional
conditional to sample an expression are surprisingly pretty good.

If you add generative testing via spec or test.generative, you can simulate
lots of the scenarios you’d otherwise be desperate to have a dump of the
environment to inspect. If you get there, profilers usually tell the story in
adequate detail. Profiling and debugging are related and often share
responsibilities.

------
henrik_w
I prefer good logging to debuggers.

Often, you can't attach a debugger to the production system, but you can get
logs. Also, you are often interested in the whole sequence of events leading
up to the problem, not just the current state. And, to find where to look in
the first place, you need to get an idea of what's wrong - a debugger doesn't
help with that.

[https://henrikwarne.com/2014/01/01/finding-bugs-debugger-
ver...](https://henrikwarne.com/2014/01/01/finding-bugs-debugger-versus-
logging/)

~~~
emsy
Those are orthogonal to each other and you should use both. But in
development, a good debugger with data breakpoints, logging and conditional
breakpoints is much more focused than an overly verbose log.

~~~
henrik_w
It's not overly verbose, and it's something you need anyway.

~~~
emsy
Overly verbose in the context of debugging. Logs almost can't be too verbose,
because as you said, they should be able to tell you the sequence of events
that lead to the error.

------
gnufx
Concerning making money with proprietary debuggers, DDT (no, not that one) and
Totalview are very expensive parallel debuggers of longstanding. Totalview has
some form of reversible debugging and is unusual in not using GDB. They're
actually rarely used in my experience, for various reasons -- not just because
they're un-affordable for most sites.

------
ensiferum
I find that the problem is that the better you become at engineering (i.e.
designing and writing your code) the worse you become at debugging. This is
because in a well designed system you have many (as many as possible)
independent components that you can look at one piece at a time and you've
written the appropriate unit tests for them. And subsequently you don't spend
that much time in the debugger. In my experience this way most bugs
(integration bugs aside) are rather trivial. In my own projects (over the past
+15 years) most of my debugging involves just running gdb on my unit test
until a test assertion fails, setting a breakpoint _before_ the failed
assertion and rerunning so that I can step through the computation.

On another note there's actually this thing I'd like to call "debuggability".

I find that if you just thinking of this as a measure of how easy your program
is to debug and when making design decisions you think of "which design makes
this thing easier to debug" your program will be simplified. To me mean high
"debuggability" at design level means cohesive independent components which
makes it easy to write unit tests for them which then subsequently makes it
easy to debug them as well.

Other thing is sometimes I see people write code like this:

    
    
      getFoo()->getBar()->doComputation();

or something like

    
    
      doComputation(getFoo()->GetSausage());
    

And depending on your debugger this can be super inconvenient to debug
through. For example with gdb trying to step into any specific function
basically requires you to find that function and set a breakpoint there.
Otherwise you'll have to go through a lot of unrelevant stuff (especially if
you're for example using std::unique_ptr)

If you used some temporary variables to store the results of those function
calls again you'll make your program more "debuggable".

~~~
sfink
> For example with gdb trying to step into any specific function basically
> requires you to find that function and set a breakpoint there.

step, then finish. Repeat until you're in the right subexpression.

Step will enter a subexpression. If it's not the one you want, finish will run
until it's done. Repeat.

------
slavik81
My personal pain debugging C++ with GDB recently has been the difficulty of
creating objects. I spent a long time trying to figure out how to call
operator[](vec2u) on my custom data structure.

In the end, I gave up and built self-introspection tools directly into the
program itself. It may not be a general solution, but it worked for me.

------
rhinoceraptor
There's another class of debugging tools: tracers such as dtrace and bpftrace.
There are a lot of bugs that you just can't reproduce in a debugger, or even
in a development environment at all. You need to be able to debug in
production safely and without huge performance penalties.

~~~
non-entity
I actually looked at learning dtrace recently, wondering if it could help me
with a handful of isuses, but so far I've been unable to find information
related to its capabilities, or up to date guides / books/ tutorials

~~~
rhinoceraptor
Brendan Gregg's website is a good resource, for both dtrace and ebpf:

[http://www.brendangregg.com/dtrace.html](http://www.brendangregg.com/dtrace.html)

------
Kapura
I use the Visual Studio debugger on an almost daily basis, and it'll be hard
for me to use another tool if/when I leave my industry. Conditional and data
breakpoints are gamechangers.

To be frank, I didn't use debuggers much at all throughout my tenure at
university, my internship at Amazon, and my first few gigs in the games
industry. I still had bugs, but I was able to solve them mostly through
printf, gut checks, and dousing rods.

But when I got to AAA console game development, the leads had to show me how
to use the immense power of the VS debugger, and now I have difficulty imaging
life without it. Different scales of problems require different tooling to
manage.

------
seanmceligot
Break-point debuggers take too long in most environments (with a few
exceptions). I use linters, compilers and type checkers as my first line of
defense. Next, logging and unit tests. Lastly, I like to have in interactive
command line REPL with good tab completion if one is available and not to
difficult to setup.

In python, for example I use: pycodestyle, pyflakes, mypy and logging, and
bpython.

For web apps, reload time is the most important factor. Work on reducing the
time to compile, reload and test.

~~~
roca
But have you used a really good debugger, e.g. a record-and-replay debugger
that lets you jump back in time to where some selected variable was modified?

The whole point of the OP was that just about all debuggers suck, but
debuggers can be a lot more powerful.

------
timwaagh
Second time I saw an ad for this product on this site. Don't get me wrong. I
like the idea of innovation in this space. But I think they should just buy
advertising space.

------
mc3
Unit tests make debugging suck a bit lest, because of the ability to change
the test case (on local machine only!) / and tested code iteratively in
combination with the debugger to find the source of the problem, with fast
feedback cycle. Debugging using the UI is another tool. Advantage is you can
quickly cover new scenarios and change what you do based on new info. Downside
is loading time can be a lot longer than running a unit test.

------
StillBored
The problem with runtime debuggers is that they are only really useful in a
development environment. Outside of that you have to rely on core dumps,
logging and various other post analysis tools.

Which is why projects have to built the debugging into the product in one form
or another. A topic that is far to broad/long for a posting here, but most
projects end up implementing in various forms if they live long enough.

~~~
roca
With record-and-replay debugging that is not true. People can and are
capturing failures outside the development environment (e.g. in CI) using rr
and debugging those recordings in Pernosco, for example.

------
theamk
> In particular, developer culture is that developers don't pay for tools,
> especially not debuggers.

I blame larger programming teams. It’s a chicken and egg problem: if you want
to start using a commercial debugger, you need to buy it for everyone on dev
team, so you can share your knowledge and benefit from others’ help; and you
are not going to buy a debugger for everyone if you are not using it yourself.

------
PretzelFisch
Posts like this reminded me how spoiled I am now with the visual studio
debugger for c#. I think it's still one of the best in the industry.

------
toolslive
you used to have them: Borland had a good IDE with a good debugger for Pascal,
C, ... (even Prolog) in the early 1990s. IBM's visual age (Smalltalk, Java)
was great too: stop in the neigbourhood of the bug, edit the code, and
continue without restarting...

Now we're exporting logs from a docker in docker on travis. O tempora O mores

------
howard941
Sometimes just having a debugger is nice. There's an entire ecosystem based on
cheap Arduino boards but the Arduino (teensy, anyway) intentionally hides its
debug SWD pieces away to prevent pirating, leaving the developer high and dry
with nothing but printf debugging.

------
ipoopatwork
It's probably the only part where hardware development (FPGA or ASIC) has a
significant edge: "time travel debugging" is the norm, where you have a wave
window with your design state cycle by cycle. Can't wait to see more of that
in software!

------
sesuximo
FWIW gdb can record (sometimes)
[https://sourceware.org/gdb/wiki/ProcessRecord/Tutorial](https://sourceware.org/gdb/wiki/ProcessRecord/Tutorial)

~~~
sfink
Plan to spend a lot of time first narrowing down the time window of interest
so it doesn't overflow its recording buffer, and then waiting for it to run at
1/100 speed.

I used to use it. It was so painful and limited that I think it helped only a
single time, and was all in all a net loss in productivity.

------
xanth
Has anything like RR been developed for the .net Core CLR or Framework?

~~~
james_s_tayler
I was under the impression the Enterprise version of visual studio includes a
debugger that functions in a similar manner. Capture the entire execution and
allow it to be replayed and shared with others.

~~~
imstuff
It seems it's only for Azure: [https://docs.microsoft.com/en-
us/visualstudio/debugger/debug...](https://docs.microsoft.com/en-
us/visualstudio/debugger/debug-live-azure-virtual-machines-time-travel-
debugging?view=vs-2019)

------
loupeabody
just gonna leave this here RemedyBG for Win64:
[https://remedybg.itch.io/remedybg](https://remedybg.itch.io/remedybg)

------
hannofcart
> TL;DR Debuggers suck, not using a debugger sucks, and you suck.

While at the outset this post seems like a rant, I believe the opinion
expressed is very astute.

In a perfect world, we would all write programs as a composition of mostly
small pure functions, with type systems that can enforce provable correctness,
and a small part of the code would generate 'effects'.

We don't live in such a world.

We often inherit gargantuan codebases written in an extremely imperative
fashion, with a lot of state management baked into functions thousands of
lines long with conditional and loop blocks nesting 5 levels deep.

And there's a bug in one of those loops.

A good step debugger that we could reverse step and inspect would save us
somewhat in such times of perils. But they don't really exist. Or if they do,
don't integrate with your editor.

So needing debuggers suck. Debuggers themselves suck. Developers suck.
Deadlines suck.

Sigh.

~~~
sanxiyn
The whole point of this post is that such a debugger exists, and that you can
use it, right now.

------
C1sc0cat
Try working in older languages with no real debugger that only has print
statements

------
The_rationalist
Is there something similar to RR for Java?

~~~
roca
Yes, I think Chronon is still available:
[http://chrononsystems.com/products/chronon-time-
travelling-d...](http://chrononsystems.com/products/chronon-time-travelling-
debugger)

------
orisho
Disclaimer: I'm a senior dev at rookout.com, which does something similar to
Pernosco/rr but tailored to languages used for backend/web like Node.js, Java
and Python instead of statically compiled (to machine language) languages like
C/C++.

I definitely agree - I've watched people painstakingly stare at code, add
print/log statements and waste enormous amounts of time -- and then I asked
them why they don't just use a debugger and always got unsatisfactory answers
that seem to hide the fact they have a hard time using the tool effectively.

Of course, like you say, using a debugger effectively in many settings is
difficult. A multi-threaded program setting a breakpoint and inspecting state
difficult, as finding the right thread to inspect can be difficult -- not to
mention having other threads alter your debugger's display state by breaking
once you're already inspecting the correct thread. I've had this happen with
native software (e.g. C++), for which rr is great (but unfortunately I've
worked mostly on Windows), but also for backend software written in higher
level languages such as Python.

Using a classic debugger effectively in a multithreaded environment requires
being aware of the issues you're about to encounter, and knowing how to deal
with them with the debugger's tooling -- which ranges between difficult and
not possible. For example, with Windbg - it is possible to break on the
correct thread, but setting a conditional breakpoint (even a simple one) is
nothing less than an incantation to someone who's never used it before. As
such, the cost of entry is very high, and I've watched kernel driver
developers use prints and asserts to debug.

While at my previous job, I thought that logs are a really crappy way of
debugging an issue in production when it arises - if you have all the right
information and can fetch it quickly, great. But often, that's not the case -
you might end up needing to add logs, or worse - the issue will be in a
hotspot where you simply can't log - since you'll have _too much_ data, or you
can't log everything for compliance reasons.

I had a similar idea to Rookout's (and even almost accidentally raised money
just by talking to the right people, although I was not prepared at the time
to start a company), and discovered Rookout later on, going on to use them.

When I left my last job, I joined Rookout because -- like O'Callahan (and
Pernosco) -- I'm very much a strong believer that the way debugging is today
isn't good enough, and can be a lot better.

Nowadays Rookout's product for interactive debugging backend apps is pretty
much complete, works well and fast -- and we're exploring other ways to make
debugging easier.

Rookout's a bit different to rr/Pernosco in that it allows you to collect
variables from anywhere during runtime (instead of allowing you to replay a
recording) -- it's a bit similar to allowing you to add a log that logs
locals() anywhere, except without changing your code, redeploying, etc.

------
randomsearch
Just came here to say - JetBrains IDEs have fantastic debugging.

------
forgotmypw3
Possibly controversial opinion: The best debugger I ever used was VB3-6, and
same-era VBA.

------
aleister_777
You're probably right, but no one will believe you. Sorry.

------
verdverm
Since I switched to Golang as a primary language, I have not needed a
debugger. Prints, logs, and stack traces on exceptions has been sufficient.

------
nojvek
Author goes on a big rant and says rr solves everything. Big claim with little
backing.

There’s tons of languages. The closest I’ve seen to a sane debugging
experience is vscode. It works the same across languages. Python, browser js,
node js, go, Java etc

Mostly because they all share same underlying debug adapter protocol.

Now if you truly want to make debugging phenomenally better. Get those new
apis in the protocol for reverse debugging and the likes. Then make it happen
across a bunch of languages. I will pay It. I know most people will.

We just want something uniform that works

Log debugging is easiest. Why ? Because it’s simple AF. Running a 1000 things
in parallel. Aggregate the logs and filter what you need.

Going through layers of virtual computation? Physical -> Node pool -> service
-> pod -> container. Write to log, aggregate.

There’s been a lot of promise but haven’t seen anything that beats the
simplicity of structured logs + good log analyzer tool.

