
A Visual Debugger for Jupyter - sandGorgon
https://blog.jupyter.org/a-visual-debugger-for-jupyter-914e61716559
======
cs702
This animated GIF shows the new debugger in action, allowing point-and-click
step-by-step execution at breakpoints and easy inspection of variables, call
stack, and source code:
[https://miro.medium.com/max/1400/1*NP0bYBdrhwgpJpKDhPLWrQ.gi...](https://miro.medium.com/max/1400/1*NP0bYBdrhwgpJpKDhPLWrQ.gif)

It looks to be a very welcome improvement for those of us who routinely use
Jupyter notebooks for "REVL" (read-eval-visualize-loop) experimentation.

Has anyone here had a chance to test how well this works with objects from
Python's scientific stack such as numpy arrays, pandas dataframes, matplotlib
objects, PyTorch and TensorFlow CPU/GPU tensors, and so on?

------
cfv
Been wondering about this for a while... How does one "graduate" a notebook to
an actual program on a server? Is there an actual standard procedure for that?

I'm pretty sure that if your notebook is big enough to debug what's in it
you'd be doing everyone a disservice keeping it contained like that.

~~~
realbarack
The fast.ai folks wrote nbdev to help with that.
[https://github.com/fastai/nbdev](https://github.com/fastai/nbdev)

------
Gehinnn
I don't see how this debugger is visual, but anyway it's cool to see that
jupyter finally has a debugger.

This is what I would call a visual debugging experience:
[https://github.com/hediet/vscode-debug-
visualizer/blob/maste...](https://github.com/hediet/vscode-debug-
visualizer/blob/master/extension/README.md)

(Disclaimer: I'm the author of that extension)

~~~
throwqwerty
cool extension but literally every other use of the term visual debugger
refers to a graphical interface for something like gdb (where you have to
manually type in commands to set break points, step, continue, etc). so I
don't think it makes sense to claim that this is not a visual debugger simply
because you've decided to redefine the term to suit your needs (publicizing
your extension).

~~~
Gehinnn
I guess, growing up with Delphi IDE, Visual Studio, Browser Dev Tools and
finally Visual Studio Code, all calling their "visual debugger" just a
"debugger", I got used to debuggers being "visual" by default.

As Jupyter has very powerful visualizations, I initially thought they somehow
integrated their visualizations into the debugger, just to see that it's an
ordinary debugger every modern IDE has. I know a debugger and it's UI is a
crazy complicated beast, but I would expect any modern programming language to
have such a debugger.

Just because there are text-based browser like lynx, modern browsers shouldn't
start calling their products "visual browser", but I get your point.

Sorry for publicizing my extension here. It's free and open source, works with
Python and might help a lot of people. According to github insights, traffic
decreases whenever I don't publicize it somewhere.

~~~
lonelappde
You're not wrong, but Jupyter is terrible for everything about programming
_except_ in-document HTML/SVG visualizations of expressions (no support for
unit testing, laggy Ui response to user input, broken Undo/Redo that throws
away data, buggy browser based text editing), yet that one thing is worth all
the suffering, so we'll take any improvement we can get.

I don't understand why VisualStudio doesn't have LightTable like expression
playground yet, but until it does, Jupyter is what we have -- a fancy REPL and
document publishing format being abused as an IDE.

------
mxcrossb
I always thought a debugger was more useful for when you got a big buried in a
long execution sequence and produced by unknown combinations of variables.
Where as usually my notebook cells are only a few lines long, and with a very
clear state going in. Can any of you speak to the practical need for a
debugger in Jupter?

~~~
Hendrikto
> Where as usually my notebook cells are only a few lines long, and with a
> very clear state going in.

That is only true in one very specific case: When you are executing all code
cells in order, from top to bottom, exactly once.

If you start executing cells multiple times, or out of order, or even worse,
execute only parts of cells (which is possibke in many Jupyter UIs), all bets
are off. Anything can happen.

------
kbumsik
I haven't used Jupter since I switched to web dev, but it's greate to see
Jupyter's debugging experience gets better.

BTW, it looks pretty much like Spyder. AFAIR Spyder is a funded project by
Jupyter. What happened between two projects?

~~~
mattip
Spyder was funded by Anaconda for a while, and is now funded by NumFocus and
Quansight [https://github.com/spyder-ide/spyder/wiki/Current-Funding-
an...](https://github.com/spyder-ide/spyder/wiki/Current-Funding-and-
Development-Status)

------
westurner
Source: jupyterlab/debugger
[https://github.com/jupyterlab/debugger](https://github.com/jupyterlab/debugger)

------
anonu
When will jupyter have "highlight and execute" functionality? The cell concept
is fine, but I'm constantly copy pasting snippets of code into new cells to
get that "incremental" coding approach...

~~~
westurner
So, I went looking for the answer to this because in the past I've installed
the scratchpad extension by installing jupyter_contrib_nbextensions, but those
don't work with JupyterLab because there's a new extension model for
JupyterLab that requires node and npm.

Turns out that with JupyterLab, all you have to to is right-click and select
"New Console for Notebook" and it opens a console pane below the notebook
already attached to the notebook kernel. You can also instead do File > New >
Console and select a kernel listed under "Use Kernel From Other Session".

The "New action runInConsole to allow line by line execution of cell content"
"PR adds a notebook command `notebook:run-in-console`" but you have to add the
associated keyboard shortcut to your config yourself; e.g. `Ctrl Shift Enter`
or `Ctrl-G` that calls `notebook:run-in-console`.
[https://github.com/jupyterlab/jupyterlab/pull/4330](https://github.com/jupyterlab/jupyterlab/pull/4330)

"In Jupyter Lab, execute editor code in Python console" describes how to add
the associated keyboard shortcut to your config:
[https://stackoverflow.com/questions/38648286/in-jupyter-
lab-...](https://stackoverflow.com/questions/38648286/in-jupyter-lab-execute-
editor-code-in-python-console)

------
Myrmornis
I'm a programmer, mostly not a data scientist nowadays, mostly working with
Python. I have tried Jupyter Lab/Notebook on and off over the last 10 years,
and I believe I have now firmly settled on my conclusion:

Everyone should aim to minimize the amount of work they do in Jupyter Lab /
Notebook.

It shocks me a bit to find myself saying that, as it is such a beautiful piece
of work. Furthermore the people who wrote it are better software engineers
than I'll ever be: the frontend, the zeromq-mediated communication with the
kernel, the fact that the architecture has generalized so successfully to
other language kernels, its huge popularity and reach. Nevertheless, I believe
I'm serious. It really comes down to just two related issues, but they're
extremely important: debugging and version control.

If you're a software engineer, and not a data scientist, here's how you
probably debug already, or if not then how you should debug:

\- You identify (a) commit(s) on which the behavior is correct, and (a)
commit(s) where it is not correct.

\- You experiment with fixes. Perhaps you stash them, perhaps you create
experimental commits.

The critical point is that you use your version control system (probably Git)
to navigate between alternative versions of the code. With a single command,
you can switch the version of your code base, and the subsequent process you
invoke to test your code is a fresh process, unpolluted by any state from the
version of your code that you were on 30 seconds ago.

In contrast, Jupyter notebook does not encourage this style of work at all. In
practice, what you will do when trying to debug some code in Jupyter is
comment out lines, temporarily delete code, add experimental lines, add
experimental new cells, etc. All creating a working tree, and a collection of
in-memory python objects, that is a baffling mixture of changes related to the
original feature development, and changes related to experimental debugging.
Debugging will wear you out, as the state of your notebook gradually
approaches complete incomprehensibility.

If you're a software engineer, you'll already know the benefits of being able
to make precise adjustments to the state of your code with git commands. You
want to learn statistics and data analysis skills from data scientists, but in
doing so you should not regress to a worse style of development by starting to
write much of your code in Jupyter notebooks.

And if you're a data scientist, you will want to acquire the debugging skills
of software engineers. If you are not using git, you want to start learning it
now.

Crudely, we can imagine a 2-dimensional diagram with one axis for engineering
skills and another for data science skills. Everyone wants to be in the top-
right quadrant. In that quadrant, version control is used, and the version
control system is used for debugging. Debugging is rather important in
developing all software, whether scientific/numerical or not.

So both groups should be minimizing the amount of code written in the Jupyter
notebook UI: instead, write code in a standard Python package, in a
virtualenv, installed in editable mode with `pip install -e`. If you need to
use a notebook for graphical display, or HTML display of Pandas dataframes, or
display of an audio playing widget, or any of the other amazing things it does
so well then fine: use importlib.reload in your notebook to load and reload
the bulk of your code from your Python package. The notebook should just
feature calls to plotting routines etc that you have implemented in standard
code files using your text editor/IDE. You could even aim for your notebook to
contain so few lines of code that in some projects you might not even bother
committing it.

~~~
dorfsmay
Are there projects to integrate notebooks with version control?

~~~
Myrmornis
This is the most promising one I'm aware of. I tried it in the course of
trying not to accept the conclusion I outlined above.

[https://github.com/mwouts/jupytext](https://github.com/mwouts/jupytext)

------
alfla
NICE!

------
throwqwerty
I'll take this opportunity to get jupyter advice: one thing I don't understand
about jupyter vs jupyterlab is why notebooks in jupyter classic have
functioning vim bindings (ie in cells) through the codemirror extension but
jupyterlab notebooks do not. in jupyterlab you can have vim bindings in the
text editing view but not in the notebook view. for the life of me I cannot
understand this design decision. how hard can it be to just leave whatever
thing that makes it work in jupyter classic alone so that it continues to work
in jupyterlab.

~~~
lp251
there is an extension to provide vim bindings within cells- jupyterlab-vim

~~~
throwqwerty
yes there is but somehow it is not as good as the codementor extension (iirc
you it doesn't play well with cell navigation bindings)

~~~
lkirk
Can you elaborate a little here? I made the switch from classic notebooks to
jupyterlab recently and find the bin experience very similar (there's a few
subtle differences that I can't remember offhand, but I don't recall having
any problems with cell navigation)

------
arnsholt
This looks nice and all, but on the other hand it's kind of depressing how
people are expending all this effort on Jupyter and surrounding stuff, just to
make something that almost, but not quite as good as Smalltalk.

~~~
mutant
Would you expand on this? I don't know smalltalk, and I'm interested in what
you see as better?

~~~
xkriva11
It uses the fusion of the IDE and the program you develop. You have vast
possibilities of objects visualizations (inside a debugger or in separate
windows), you can select some part of the code inside the debugger and let run
it in separate debugger etc. See
[https://pharo.org/features](https://pharo.org/features)

