
What’s wrong with computational notebooks? - ashort11
http://web.eecs.utk.edu/~azh/blog/notebookpainpoints.html
======
azhenley
Co-author of the study here. Let me know if you have any questions or how you
overcome some of the problems we identified!

~~~
snapetom
I just wanted to say thank you. Many of the points in your study strikes a
nerve. Part of my responsibility at my last job was to introduce good software
engineering practices. What happens? The data scientists go rogue and start
running notebooks left and right. How do they productionize their work? Well,
they don't. They were academics. All they know is that the models ran fine in
their notebooks on their laptops. Meanwhile, we didn't have anyone that was
devoted full time on model productionization.

Sharing data? They had enough problems sharing their notebooks.

~~~
jasode
_> to introduce good software engineering practices. What happens? The data
scientists go rogue and start running notebooks left and right. How do they
productionize their work? Well, they don't. They were academics. _

My background is programming (instead of data analysis & modeling) so I'm
sympathetic to your idealistic "software engineering" view... but I'm also
sympathetic to the academics' side as explained by Yihui Xie's blog post:

[https://yihui.org/en/2018/09/notebook-
war/](https://yihui.org/en/2018/09/notebook-war/)

He's convinced me that criticizing non-programmers for using (or over-using)
computational notebooks when it should be a "proper" programming language and
deployment is like criticizing financial analysts over-using Excel to learn
how to program VB or Python and re-write their spreadsheets into a "proper
database" like Oracle or MySQL. That's just not reality. This divide between
"end user tools" and "proper programmer tools" will _always_ exist because
there is _no perfect tool in existence_ that serves the needs of both skill
sets. Therefore, the programmers will always be able to say the data
scientists or financial analysts are "doing it wrong".

~~~
BeetleB
> He's convinced me that criticizing non-programmers for using (or over-using)
> computational notebooks when it should be a "proper" programming language
> and deployment is like criticizing financial analysts over-using Excel to
> learn how to program VB or Python and re-write their spreadsheets into a
> "proper database" like Oracle or MySQL.

I think this is very much off the mark. For sure plenty of scientists are poor
programmers, but that isn't the reason they use notebooks. It is because:

They are not attempting to write something that will run everywhere, and
often. They are either analyzing some data or doing rapid prototyping. For the
latter, it's like criticizing someone who uses a REPL. It's just that the
Notebook is much more powerful than a simple REPL that one can safely stick to
it. Imagine you will do 40-50 prototypes and only _one_ of those may end up
worthy enough to make a product out of, and you don't know which one that will
be. If you used a non-notebook environment, you'd give up in frustration by
the time you hit the 15th one.

As you said: At the moment, there simply isn't an alternative that allows for
rapid prototyping _and_ is production ready. It's a hard problem to solve -
there's a reason no one had solved it for decades (well before notebooks were
a thing).

Had notebooks not been invented, you would have the same people handing you
MATLAB code asking you to productize it.

Claiming they are beginners/novice programmers is off the mark. Peter Norvig
started using notebooks for a reason, and no one would call him a novice. I do
SW for a living, but when I need to analyze data and visualize it, I'll pick a
notebook over "proper" SW tools any day.

------
moultano
I want a notebook where causality can only flow forward through the cells. I
hate notebook time-loops where a variable from a deleted cell can still be in
scope.

1\. Checkpoint the interpreter state after every cell execution.

2\. If I edit a cell, roll back to the previous checkpoint and let execution
follow from there.

I can't tell you how many times I've seen accidental persistence of dead state
waste hours of people's time.

~~~
fryguy
My problem with notebooks is that I feel like the natural mental model for
them is a spreadsheet mental model, not a REPL mental model. Under that
assumption, changing a calculation in the middle means that all of the cells
that depend on that calculation would be updated, but instead you need to go
and manually re-run the cells after it that depend on that calculation (or re-
run the entire notebook) to see the effect on later things. Keeping track of
the internal state of the REPL environment is tricky, and my notebooks have
usually just ended up being convenient REPL blocks rather than a useful
notebook since that's the workflow it emphasizes.

~~~
WorldMaker
That's something that I think Observable [1], in my modest usage, seems to do
well.

[1] [https://observablehq.com/](https://observablehq.com/)

~~~
jacobolus
Yep, the real complaint is “dead state”, not out of order execution. Worrying
about linear flow per se turns out to be misguided based on lack of
imagination for/experience with a better model: reactive re-rendering of
dependent cells. Observable entirely solves the dead state problem, in a much
more effective way than just guaranteeing linear flow would do.

* * *

More generally, Observable solves or at least ameliorates every item in the
linked article’s list of complaints. (In 2020, any survey about modern
notebook environments really should be discussing it.)

I found the article quite superficial. More like “water cooler gripes from
notebook users we polled” than fundamental problems with or opportunities for
notebooks as a cognitive tool. I think you could have learned more or less the
same thing from going to whatever online forum Jupyter users write their
complaints at and skimming the discussion for a couple weeks.

I guess this might be the best we can hope for from the results of a
questionnaire like this. But it seems crazy having an article about notebook
UI which makes no mention of spreadsheets, literate programming, Mathematica,
REPLs, Bret Victor’s work, etc.

From the title I was hoping for something more thoughtful and insightful.

~~~
SiempreViernes
You can get a jupyter extension[1] that allows you to add tags and
dependencies and this way construct the dependency graph as you go along. Of
course, you have to do it manually and the interface is a bit clunky, but it
does what it says.

In practice I think taking care not to accidentally shadow variables is much
more important: this dependency business only makes sense once you have a
clear idea of what you need and by that point you are mostly done anyway.

[1] [https://jupyter-contrib-
nbextensions.readthedocs.io/en/lates...](https://jupyter-contrib-
nbextensions.readthedocs.io/en/latest/nbextensions/execution_dependencies/README.html)

~~~
jacobolus
I don’t understand what you are trying to say in your second paragraph, but I
highly recommend you spend a few weeks playing with
[http://observablehq.com](http://observablehq.com) instead of speculating
about the differences.

In practice, I find it to be dramatically better than previous notebook
environments for data analysis, exploratory programming / computational
research, prototyping, data visualization, and writing/reading interactive
documents (blog posts, software library documentation, expository papers ...).
It has a lower barrier to starting new projects, a lower-friction flow
throughout

I find it better at every stage of my thinking process from blank page up
through final code/document, and would recommend it vs. Jupyter or Matlab or
Mathematica in every case unless some specific software library is needed
which is unavailable in Javascript. The only other tool I really need is pen
and paper, though I also use
[http://desmos.com/calculator](http://desmos.com/calculator) and Photoshop a
fair bit.

------
andrew_n
I used Mathematica’s notebook interface quite heavily 15-20 years ago;
Jupyter’s interface is a clone of that in many ways.

At the time, my workflow was to use two different notebooks for everything:
foo.nb and foo-scratch.nb. I’d get things working a piece at a time in foo-
scratch.nb, not caring at all how it looked, not having to worry about leaving
extra output or dead ends of explorations lying around; then the refined cells
would be copied over to foo.nb, which would get pristine presentation, and
which I could run top-to-bottom.

This workflow worked pretty well for me: very clean reproducible output, with
the ability to easily refer back to all the steps of how I’d derived
something, along with copious detailed private notes.

I never had to use it but I’m pretty sure each cell even had its modification
time stored in the metadata in case I wanted to view a chronological history.

~~~
zneveu
I make a "scratch pad" section of my notebook and work on ideas there. Then
once I've pieced together a function line by line and tested it a bit I move
it up to where it should be in the chronological order of the notebook. Kind
of like your two notebook system but makes copying easier in Jupyter.

~~~
EForEndeavour
I do the same, though it feels dangerous because both the good-copy and
scratch sections share the same kernel. JupyterLab works on .ipynb files, and
makes it way easier to copy (or drag and drop) cells between different
notebooks. One of these days, I plan to switch to JupyterLab to get a sense of
what else it offers above Jupyter Notebook.

------
hprotagonist
See also Joel Grus' talk, "I Don't Like Notebooks":
[https://www.youtube.com/watch?v=7jiPeIFXb6U](https://www.youtube.com/watch?v=7jiPeIFXb6U)

slides:
[https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUh...](https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUhkKGvjtV-
dkAIsUXP-AL4ffI/edit)

~~~
funklute
Superb talk! It's worth noting that a lot of the issues he brings up,
ultimately stem from the format in which Jupyter notebooks are stored. R
notebooks, with their plain-text stored format as well as code-chunk
parameters, solve some, but not all, of these problems.

~~~
hprotagonist
[https://github.com/mwouts/jupytext](https://github.com/mwouts/jupytext)

doesn’t solve the state management testing or tooling issues though, but
commits are slightly less awful.

~~~
funklute
Yea, when I first found that extension, I was pretty excited about it. But it
ultimately is not a first class citizen in the way it is for R notebooks, so I
simply don't feel as comfortable using this as I might otherwise have been.

------
rcar
I'm someone who has been programming for a very long time and has been using
notebooks for a reasonably long time (and almost always starts projects with
them), my feeling is that they are a bit like C in that they make it easy to
accidentally shoot yourself in the foot if you aren't careful. I always strive
to end up with a notebook that can be "Run All" from a fresh clone, and I'd
say that I'm successful with that maybe 60-70% of the time, and am close
enough that I can fix it in the remainder.

As the article (and the many others like it that have frequently cropped up as
soon as IPython Notebooks first started ramping up in popularity) points out
though, a lot of newer users don't have the discipline to ensure that they're
not jumping around too much. It's not a problem for them in the immediate term
since they know how the state ought to work, but then it becomes a mess when
they try to share it with someone else (or to run it themselves again 3 months
later).

The challenge though is that the data analysis workflows that it allows are
unbeatable by any other tools I've tried. In the end, it may just be that it's
the worst form of data programming except for all of the others that have been
tried.

------
bryanhpchiang
I interned @ Google AI last summer; used notebooks nearly everyday. Estimated
productivity gain is 3-5x.

Biggest tip I have is to turn auto reload on, then write the bulk of your code
as modular functions and call functions within your notebooks. Keeps the
notebook tidy and it’s easier to push your code this way.

It’s also easier for sharing since most people viewing your notebooks
(mentors, people outside your team) are interested in results/artifacts such
as metrics, generated text, images, audio, which notebooks display well (not
your code).

~~~
randomsearch
Beware perceived gains that (1) benefit you at the detriment of others or (2)
have hidden costs exposed at a later stage.

------
stared
(A frequent Jupyter Notebook user here. For data exploration, and teaching
deep learning - then Colab is indispensable.)

The main question is: what are the alternatives, for data exploration (and
sharing its results). Similarly, for data science tool demos, Notebooks shine.

IMHO the problem is not in the notebooks, but in how they are being used (i.e.
the workflow). By writing scripts in py files, and using notebooks only to
show their results (processed data, charts, etc) we get the best of both
worlds.

The only build-in problem with Jupyter Notebooks is JSON, mixing input and
output (and making it pain to work with version control). But here RMarkdown
(and a few other alternatives) work well.

~~~
fhennig
Yes, the article mentions users copy pasting snippets from their personal
"library". Well, that could just be made into an actual library of functions
to call.

I'm currently at uni enrolled in an AI/ML degree, and there are a lot of
people with no previous exposure to programming. It's just that most people
don't know that these things are possible, don't want to learn another tool
(IDE) and are not interested in longevity of the code, just in the results.
This shouldn't sound like me complaining, I totally understand. I think a lot
of the stuff could be solved with just better tooling, but a familiarity with
software development is definitely helpful.

Also a while back streamlit
([https://www.streamlit.io/](https://www.streamlit.io/)) was here on HN and
since then I've been meaning to try it. I think this could be a good approach
to bring together the best of both worlds.

~~~
sillysaurusx
_most people don 't know that these things are possible, don't want to learn
another tool (IDE) and are not interested in longevity of the code, just in
the results._

This approach is arguably more effective than wasting time trying to refactor
everything into a library of functions.

Programming-as-crafting needs to be more of a thing. Not everything is written
to be long-lasting. Even HN ushered the ugly code into hook functions that
weren't shipped with the main codebase.

------
comment_guy
I don't get why anyone one who knows how to use an IDE would ever use a
notebook, the coding experience is garbage in comparison. I understand they
started as a way to get STEM kids coding quick, but now they are like a
standard in data analysis and data science, with those people needing
experienced devs to translate the notebook into production code. This just
drives the silo walls up higher.

~~~
grp000
Doing data science in an IDE would be terrible. With a notebook, you get the
chance to load the data, view it, clean it where needed, view it again,
analyze it, model it and do anything else you need to it. An IDE means that
you can't use the previous output to guide your next operation in a direct
fashion like you can with a notebook.

~~~
bllguo
as a counterpoint, plenty of R folks are pretty happy doing all of that in
Rstudio

~~~
meztez
It is interesting to see this discussion about notebooks while I'm thinking
about all the RStudio users who do all their work inside the IDE and are
pretty happy. Notebooks seem like such an inferior tool to me. I'm also
extremely bias.

------
teekert
I don't know what this document is meant to do but you will have to take my
Jupyter-lab instance from my dead cold hands.

I love notebooks, I work fast, line by line I execute commands and I
immediately see the output (dataframes or graphs). For complex code I have an
editor open (in jupyter-lab or vscode) for some functions and classes. But the
main developing is done in the notebook, anything that ends in a module start
in my notebooks.

As a biologist that learned to program after 30 I just don't understand how
you can develop data processing code without such a close handle on dataframes
and without checking in graphs/visualizations if your code does what you
expect. I don't see how I would do that in pure vscode of other IDEs.

I also don't understand this sentence: "Once the data is loaded, it then has
to be cleaned, which participants complained is a repetitive and time
consuming task that involves copying and pasting code from their personal
"library" of commonly used functions." What is the alternative? Not cleaning
the code? And why copy and paste when you can perfectly fine have your own
shareable module on the side? I guess most notebook users do some kind of
hybrid development.

~~~
grenoire
Good point on the last one; I think we have to 'educate' researchers on the
fact that they can also write their own libraries and frameworks, and they
should. Even basic data manipulation utilities can be made into Python modules
and distributed at ease. If something is tedious, there _definitely_ is a way
to make it less so.

------
LifeIsBio
I love jupyter notebooks. Without them, I wouldn’t have been half as
productive as I was during my PhD.

Here’s a post I wrote just a few weeks ago describing some of the conventions
that I established for myself over the course of 5 years:

[https://jessimekirk.com/blog/notebook_rules/](https://jessimekirk.com/blog/notebook_rules/)

I suspect that a lot of the conventions I describe help mitigate problems
described here, some of which should be strictly or optionally enforced by the
notebook instead of the user.

(The site’s very much a work in progress, so expect to see odd and broken
things if you go poking around.)

~~~
jboynyc
Thanks for this! Didn't know about the watermark extension, that looks useful.

I just started working with the Guix kernel for more easily reproducible and
reusable notebooks. I suppose that's an alternative to using a conda
environment.

See here for an example:
[https://gist.github.com/jboynyc/5d0319f33e71427aa42a98c1a3a9...](https://gist.github.com/jboynyc/5d0319f33e71427aa42a98c1a3a915cb)

------
rabryan35
No mention of [https://observablehq.com](https://observablehq.com) notebooks?
They’re the best I’ve found in the “Share and collaborate” and “As products”
category. JupyterLab is still pretty great for exploratory stuff, but
visualization possibilities in observable are incredible.

~~~
azhenley
No one we interviewed or surveyed mentioned it.

------
bobbylarrybobby
The problem of notebooks has been solved by the Python extension in Visual
Studio Code (and some other editors too, although VS Code is the one I'm most
familiar with).

Editing an ordinary Python file, if you insert the comment "# %%", you turn
everything between that comment and the next "# %%" (or the end of the file)
into a code cell that can be submitted to the ipython kernel, just as in a
Jupyter notebook. The editor splits into two halves, the left half your Python
file and the right half the Jupyter notebook window with submitted code and
formatted output (e.g., DataFrames look pretty, plots display normally, etc.).
When you're done running everything, you can export the result as a Jupyter
notebook. Because you're editing an ordinary Python file, standard features
like version control and importing the file you're editing into other files
(you cannot normally import .ipynb files IIRC) work normally.

And of course since VS Code is a real editor/IDE, you can double click a file
and have it open right up (no resorting to a Terminal to start your Jupyter
session) and you get syntax themes, a built in Terminal, a git UI, code
snippets, documentation on hover, vim mode if that's your thing, etc.

The only downside I've found is that the Python extension doesn't incorporate
ipython's autocomplete in its own autocompletion, but that's a small price to
pay for getting to treat .py files as notebooks.

------
nl
The FastAI people have been working on a lot of these issues with their NBDev
too:
[https://www.fast.ai/2019/12/02/nbdev/](https://www.fast.ai/2019/12/02/nbdev/)

~~~
zneveu
Came here to post the same thing. Nbdev helps fill in the strengths that IDEs
are traditionally good at. Even if you don't use the full nbdev library and
templates, the work flow makes sense. Write code in Jupyter, export to a
python library, and you can use it everywhere else after that.

------
bloaf
So literally all of these complaints are about their particular
implementations of notebooks, not the concept of computational notebooks in
general, or are all computational notebooks destined to have unstable kernels?

In my mind, notebooks _should_ be married to a functional style of
programming, where you use the notebook's markup to thoroughly explain and
document your functions. Below your "function definition" section, you keep a
"trying things out" section where you actually plug the data into your
functions for debugging/visualizations. You can't shoot yourself in the foot
with variables because all the work is done in your function's lexical scope.
You can shoot yourself in the foot with stale function definitions, but a good
notebook interface gives you the ability to clear function definitions and run
groups of cells, so you can make sure you always run your functions in a group
that starts with a "clear function definitions" cell.

When you are done, you just cut the "trying things out" section into a second
notebook which references the functions in the first and _viola_ , you've got
a very well documented library of functions, and a new work notebook where you
can freely polish your visualizations/whatever.

------
pottertheotter
I use Jupyter Lab with Python every day. It's where I do my initial data
exploration and cleaning. Jupyter Lab is not perfect, but most of these
findings seem like they are more issues of inexperience with technology and
programming, not computational notebooks.

------
wwarner
I have been heads down in jupyter for the past couple of weeks and I finally
realized I just _DO NOT LIKE IT AT ALL_! Cracks started appearing and then
suddenly there was an avalanche of disappointment.

The first crack -- it's almost impossible to build a nice presentation in
Jupyter, because you always have to show your code and its stderr. I imported
all the TeX goodness, and it looked pretty nice, but I couldn't show the
output without showing the TeX code. Importing the TeX interpreter is quite
non-standard and means that my notebook doesn't play well with the public
servers. I also got burned by some kind of permissions issue, so that all my
charts ended up being invisible to read-only users.

The second crack -- I can only look at the code from within my own jupyter
server. The source is buried in a very noisy json format.

The third crack -- Who wants to write code in the impoverished browser based
editor provided? How many times have I deleted a closing brace that was
automatically inserted incorrectly? How can I do a global search and replace?

The fourth crack -- I can't test my code unless I include all the tests in the
notebook!

I'm complaining. I realize that I don't have anything constructive to offer,
and I'm really a beginner. However, I think some of my disappointment is
justified, as I think it was reasonable to assume that I could build my
notebooks to be next level presentations.

~~~
bb88
So once your code gets large enough that it doesn't fit neatly within a
jupyter notebook, it's time to split the code out into another package, and
then import it into your notebook.

The benefit here is that now your code and be used inside the jupyter
notebook, and also inside a webserver say.

------
tbenst
I think Atom’s hydrogen and VSCode’s python are best-in-class Jupyter clients
that achieve everything Jupyter Lab set out to do with more and better
features. I develop scripts that function top to bottom with a notebook side-
by-side that on a keyboard stroke executes code blocks from my script in the
notebook.

~~~
pqs
For those that do not know it, vscode is great for Jupyter Notebooks.

[https://code.visualstudio.com/docs/python/jupyter-
support](https://code.visualstudio.com/docs/python/jupyter-support)

------
rossdavidh
I think Computational Notebooks are a great idea, and yet I have the feeling
that we are in the process of seeing them overapplied. They are wonderful for
certain situations, and teaching or demonstrating code to others is right in
its sweet spot.

I get the impression that people are creeping in the direction of trying to do
everything with one tool, which sounds like it would end up in the same swamp
that Eclipse went into. Sometimes, you need to use different tools for
different tasks, and not everything should integrate. Just my opinion.

------
ospohngellert
I think that all the pain points of the article are a result of not using
notebooks for their purpose. In my opinion, notebooks are good for:

1\. POC/MVP: Showing that what you want to do will work before making a full
structure. 2\. Creating PDF/HTML documents with code and output. 3\.
Exploratory data analysis and visualization.

I think many of the data scientists in the article go well beyond what a
notebook is. A notebook is where you start, but should never be a production
tool.

------
evrydayhustling
This is a solid list. It will be even better if juxtaposed with current
efforts to solve each of these problems - every DS I know is addressing at
least 2-3 of these with some pet tools in their own environment. For example,
we use Panel and Holoviews to make data exploration much easier. I have a
feeling the ecosystem would improve faster if we had an index of (partial)
solutions aligned with this problem set.

One category left out of the list: testing of data pipelines (c.f. great
expectations).

------
enriquto
Jupyter notebooks are great for many purposes. They have, however, two really
tragic shortcomings:

1\. They are stored by default in stupid json files instead of plain source
code with comments.

2\. The text editing interface inside the browser is horrific and very
difficult to normalize (e.g., disable "smart" closing of parentheses, disable
the capture of classic unix copy-pasting, etc).

------
xixixao
This is a great list, and totally matches my experience. I also agree this is
solvable with tooling.

A) VS Code / IDE needs to be the primary editor B) Results are not stored with
source C) Export (build) allows packaging for whatever platform.

Python notebooks especially also use some crazy mutable APIs. In general
notebooks align with other code written by people who aren’t usually software
engineers building production systems. They’re much more about getting things
done, APIs and tools are less questioned, a lot of pain is swallowed because
PhDs have plenty of time to write a few lines of code. I don’t want to sound
disparaging towards these people, it’s just a different set of tradeoffs from
writing production grade software.

~~~
laichzeit0
Logging, monitoring, security, versioning, etc. These are things that most
often get ignored due to ignorance or inexperience, but are required for
production grade software.

------
archi42
As a computer scientist/software engineer, please allow me the question: Why
would I prefer a notebook over e.g. equivalent python script(s) in a git?

I first saw jupyter notebooks when my sister (physicist, non-programmer) used
it for analyzing economical data with pandas. Run-time for the full data set
was half a day (and IMHO for that analysis SQL would have been better suited).
I understand that as a non-programmer it looks alluring, but once the language
proficiency is build up, why not use an IDE and run the code on a shell?

~~~
PeterisP
The key factor is iteration speed.

If step A takes 5 minutes (and 5 minutes is a very short time) and I want to
experiment on step B, then I don't want to rerun step A each time while I'm
writing and running code that helps me understand what step B is going to be;
I'd want that to be interactive and immediate, not have each rerun take 5
minutes.

Storing/loading to disk is not a good option because all the data that needs
to be stored is not yet determined until the exploration is finished; If I
write code to save/load A, then I need to change (and test) it after I'm done
with B and now want to experiment with C, and it all becomes even more
complicated when I need to add an extra step and data field to step A and
rerun everything. Deciding what data should be stored in what format is
something that you can do in 'productionizing' the code _after_ you've done
the exploratory analysis.

REPL is not a good option because it's not convenient to save and replicate
the code that got you to the current REPL state.

The other aspect is that visually 'debugging' intermediary data through
various plots is not conveniently possible in IDEs. I could generate some
picture files in a folder or possibly an HTML 'dashboard' to see the results
of my most recent run but that takes extra code and effort, and the results
aren't immediately in my face like in a notebook.

~~~
archi42
Ah, I think I have a hugely different approach to data processing: For my work
I often have a very good idea what the output should look like, and what
transformations are required on the input to get there. E.g. when processing
log files to generate an overview page, or (as I'm doing right now) adding a
target to binutils (assembler, linker,...). (Obviously I'm not a data
scientist ;-)

With what you describe, intuitively I would use a library that allows me to
store&load data per step (with verifying the structure matches), or pass it
in-memory. Think JSON (yeah, slow) or something like protobuffers. That way I
could do both

> store(A(read(input)) -> file); store(B(load(file)) -> file2)

during development (or in case B is in another language as A), and in
production just

> B(A(read(some_other_input)))

But yeah, that's just _my_ intuition of course. Maybe I'd be a bad data
scientist.

However, can't you just experiment with smaller data sets? That's what I
usually do if processing is slow (e.g. instead of parsing 10GB of log files,
I'll just do 50MB to verify the processing pipeline works, and once that's it,
run it on the full 10GB and grab a coffee while it runs). Not an option for
data science?

~~~
PeterisP
Sure, if you know what the output should look like and if it's possible to
e.g. write tests to verify that it's correct, then jupyter notebooks would not
be the proper tool to use.

The intended usecase of these notebooks is in scenarios where the main output
is not the code and not a particular set transformed data, but knowledge
gained during a 'computational exploration' of that data. With that knowledge
in hand, you can then build 'productionized' code with different methodologies
(possibly but not necessarily using or adapting large parts of the code in
your notebook), _if that 's needed_ \- and in such data analysis scenarios it
often happens that it's not ever needed.

Sampling a subset of the data sometimes works. Sometimes it would alter the
results substantionally and drive the exploration in a wrong direction;
questioning and verifying assumptions is important, and it can be a big
difference if _all_ A's are also B or only 99% of them are.

------
tardenoisean
[https://datalore.io](https://datalore.io) has (1) a reactive Datalore kernel
that solves the reproducibility problem. It recalculates the code
automatically when something is changed, and recalculates only the changed and
the dependent code; (2) good completion; (3) online collaboration; (4) read-
only sharing; (5) publishing; (6) sensitive data can be saved in .private
directory that is not exposed when the notebook is shared with read-only
access

~~~
LeanderK
it seems it's cloud-based. Fun for playing around but not suitable for real
work (at least for me).

I can't just upload random data to some cloud service to work with it, also I
can't upload data if it's too big. Often the data that's valuable is very
sensitive.

------
anonsivalley652
There are these and other problems with CNs:

0\. They try to be "be-all, end-all" proprietary container documents, so they
lack generality, compatibility and embeddability. It would be better if live
code try-out snippets were self-contained and embeddable in other documents:
HTML, other software, maybe PDF, LaTex or literate programming formats. Maybe
there should be standard, versioned interpreters for each kind of programming
language in WebAssembly and cached for offline usage by the browser for
inclusion in documentation, papers, etc.?

1\. For prototyping, it is better to have try-out live code (and/or REPLs with
undo) for prototyping like what is Xcode/iOS Playgrounds for Swift or
ReInteract was for Python.

2\. Computational notebook software, that I've seen, are terrible, complex,
fragile and messy to install. The ones I've seen make TeXLive look effortless
by comparison.

3\. Beyond replicability what goal(s) are CN really trying to solve?

3.0. For replicability itself, why not have a GitLab/BitBucket/GitHub repo for
code and a Docker/Vagrant container one-liner that grabs the latest source
when built? Without a clear, consistent and simple build process, there is no
replicability, only wasted time, headaches and fragile/messy results.

3.1. Are CNs "hammers" for "nails" that don't exist?

~~~
anchpop
> Maybe there should be standard, versioned interpreters for each kind of
> programming language in WebAssembly and cached for offline usage by the
> browser for inclusion in documentation, papers, etc.

This would be incredible. Even better, the output from the code (like graphs)
should be able to be embedded in the paper. You have no idea how many papers
have errors in the code that generated the graphs/statistics/etc. and nobody
can tell because the authors rarely release the data, let alone the source

~~~
anonsivalley652
For WASM, there ought to be a package-management/registry mechanism for
installation (unless there is already? It might get complicated, but would
seem a good idea to reuse code/plugins.)... or as below, there ought to be
some caching priority mechanism.

Then for HTML assets (and CSS ones too), perhaps a hint on asset-linking tags
(a, script, link, img, audio, video, etc.) there ought to an _offline-
priority_ attribute to help the browser decide what to throw away when
clearing cache the regular way or evicting items from the cache, while being
able to leave some things deemed vital when not nuking the entire cache. Yes,
websites could be goofy and game caching mechanisms, marking everything
"vital" like for 0-pixel image cookies but I'm sure someone would make an
"RBL" (real-time blackhole list) system of which priorities on which websites
to ignore.

Related aside: There's a lot of common frameworks, libraries and bits that
could be cached user-side, with the trick either to a) herding web devs to de-
fragment their CDNs, which could create SPoF's or b) changing the standard
allowing multiple SRCs or HREFs for high-availability/less bitrot to preserve
both choice and encourage de-duplication of common assets. [0]

0\. [https://html.spec.whatwg.org/multipage/links.html#attr-
hyper...](https://html.spec.whatwg.org/multipage/links.html#attr-hyperlink-
href)

~~~
anchpop
Here's an idea - what if you could put a `hash` attribute on an script tag.
After downloading whatever it links to, the browser checks the hash matches
the one you provided. Then it could also cache the result and reuse it
whenever the hash matches, even if the link pointed elsewhere.

------
omarhaneef
Good list.

Their observations bring to mind the benefits of watching people program on
YouTube or video where you learn a style of working you may not even have
considered.

However there is one other issue that is not on the list: because a notebook
is meant to be read or shared, I always feel like my work is public and feel
less inclined to play around and just take a look at things. When I do
“transfer” my work to a notebook, it’s only surprising or interesting things
that suppress the discovery process.

------
ivan_ah
One thing that I find to be incredibly useful is the keyboard shortcut `00`
(press zero twice while focus is outside of a cell), which will restart the
kernel, clear all output and re-run the whole notebook.

This way I'm sure that "library code" that I'm editing in parallel in a real
text editor is up to date in the notebook and also solves the limits the
confusion due to run-out-of-oder problems.

The overall workflow is something like this:

    
    
      1. explore using thing.<TAB>, thing?, and %psource thing
      2. edit draft code chunk or function
      3. when chunk 80% done; move it to a module
         and replace it with an import statement
      4. press 00 to re-run everything, then GOTO step 1
    

The key to preserving sanity is step 3—as soon as the exploration phase is
done, move to a real text editor (and start adding tests). Don't try to do big
chunks of software development in the notebook. You wouldn't write an entire
program in the REPL, would you?

Sometimes I keep around the notebook as a record for failed explorations or as
a "test harness" for the code, but most of the time it's throwoutable since
all the useful bits have moved into a normal python module/script under
version control.

~~~
aldanor
Another useful tip which doesn't require always doing '00': when editing the
library code, import things like this:

    
    
        import mylib; importlib.reload(mylib); from mylib import foo
    

Then in most cases except some very entangled ones, you can simply rerun this
cell without having to restart the kernel (especially if it requires reloading
all the data).

------
breatheoften
The reality is — notebooks are and need to be developed as an app platform ...

In order to do notebooks properly — you need:

1\. discovery (Ideally static discovery) of all the state the notebook needs,
and the bulk of state the notebook will/could manipulate during its execution.
Your container needs to intercept the filesystem and the networking apis that
will be invoked so that a determination of the state that results from these
operations can be observed by the runtime and shimmed appropriately for
reproducibility and for performance optimization

2\. The notebook (and the runtime inferred model of all the required inputs)
needs to be repo stable — I Should be able to write a notebook app that reads
from the file system on my development host, deploy it somewhere, and the
runtime should take care that wherever that however that post deployment file
system read is implemented matches my local development semantics

3\. Pplatform level dependency graph needs to exist to model re-execution
requirements automatically — incorporating code changes and external state

Apple could build this And “notebook-os” would be the correct conceptual
framework for it ... anything less is always going to leave us severely
wanting

------
randomsearch
For those asking “what’s the alternative”, RStudio and Matlab already solved
the design problem (though they could be better executed).

------
desmond373
My main use for notebooks is a simple way to constantly hold a whole large
dataset in memory. That way if I want to try some feature reduction or remove
some bad result, I can just do that and not wait 10 minutes for my slow PC to
rerun my import code. I feel like an easy way to do that in base python would
draw me away from notebooks.

------
ktpsns
Obviously it's pretty hard to make general criticism of the Notebook GUI. This
is especially without comparing to a specific other user interface for data
scientists, such as a traditional REPL terminal, or some other command line
tools?

The Python world gives a good example about the sheer complexity of a notebook
infrastructure. The is IPython, there is Jupyter Notebook, JupyterLab. There
is even stuff like the SageMathCloud (nowadays called CoCalc) which is
basically a web GUI to a VPS combining command lines and various notebooks.
And hell, most of these web based interfaces try to make sharing easy.

Mabye we should start comparing these (mostly OSS) tools to the traditional
notebook GUIs of Matlab and Mathematica, something we used in the 90s and
2000s. From my feeling, they were more robust, could handle large data better,
but they lack all the tooling we get for free in the web.

~~~
nl
Jupyter Notebook/Jupyter Lab has replaced IPython as the notebook front end.

I suspect 90%+ of Python Notebook work is done in Jupyter/Jupyter Lab (or
things built on it like Google Collab/Kaggle Kernels).

 _traditional notebook GUIs of Matlab and Mathematica, something we used in
the 90s and 2000s. From my feeling, they were more robust, could handle large
data better_

I've done 10s of terabyte analysis on Jupyter (Spark backend) and I personally
know people doing petabyte work on it so this seems doubtful.

~~~
ktpsns
You probably were careful enough to understand the limits of the Jupyter
server and client (frontend).

It's easy to screw up a terminal application in data science when dumping a
large array. Many REPLs cannot handle this properly (and CTRL+C won't work).
It's easy to test this: What does your favourite notebook do when you call
some command such as (pseudocode/python here)

    
    
        print(range(int(1e7))) # or 1e8
    

In this particular example, the python CLI seems to handle keyboard interrupts
fine when the terminal (or RAM) is flooded.

------
VvR-Ox
I would love to see some gifted people using this info to further improve
tools like nteract[0].

It already eases a lot of pain you may have in comparison when setting up
jupyter notebook without the knowledge of a software developer.

[0]: [https://nteract.io/](https://nteract.io/)

------
bart_spoon
I'm not sure I understand the issue about the user repeatedly tweaking
parameters for their data visualization. If anything, that is a reason
notebooks are so nice. The repeated tweaks are due to the notebook format, its
because that's an inherent part of the data visualization process, where the
end result of a particular parameter choice is hard to predict how it will
look with a given data set. So the same process would occur whether one was
using a notebook or a script, but with a script it becomes much more
cumbersome to actually see the result. In a notebook, the parameter tweaking
for a data visualization is immediately followed by the result.

I definitely agree with most of the other points though.

------
JosephRedfern
I think it's easy to do notebooks wrong, but possible to do them right. I try
and do quick prototyping in notebook cells before moving it off to a separate
.py file, and avoid keeping any code that does anything other than
visualisation or parameter setting inside a cell long-term. That way, if you
need to run something "in production" (whatever that means in your context),
you don't end up having to pick apart and re-write your code -- you just
import the .py file you wrote along the way.

For me, notebooks are a super handy way of visualising and sharing results
during meetings, and it's difficult to imagine a more convenient alternative.

------
rb808
I tried to encourage our team to use notebooks, however everyone prefers using
PyCharm and git for sharing code. We dont have much visualization, which might
be the reason, but I was surprised just how many people just hated it.

~~~
ngcc_hk
Are you using oo? Still not sure how to “explain” an oo system once
sophisticated enough. Just better than go-to everywhere but not much. Of
course a trigger based system (gui, system) also have the same issue.

This code + explanation would not work I guess.

~~~
pjmlp
> Still not sure how to “explain” an oo system once sophisticated enough.

With a couple of UML diagrams, still the best option.

------
boomersooner
This is akin to reviewing how well a screwdriver drives nails. Yes, it has
problems. That doesn't mean it's a bad tool - you're just not using it right.
Does it require discipline? Yes, but so does the screwdriver. That being said,
I think jupyter specifically has some legacy issues around format, and I
prefer R markdown. As much as I love pycharm, it's never going to do more than
replicate the notebook experience. IMHO, the main author publishes on code
UI/UX, the title seems more like click bait. Not sure why it's so upvoted.

------
kriro
Direct link to the preprint:
[http://web.eecs.utk.edu/~azh/pubs/Chattopadhyay2020CHI_Noteb...](http://web.eecs.utk.edu/~azh/pubs/Chattopadhyay2020CHI_NotebookPainpoints.pdf)

Interesting study, I like the mixed-method approach. A quick glance at the
industry of the participants suggests that there might be a bias towards
structural data (which I think is actually acceptable as that makes up a huge
chunk of the non-academic ML-Notebook work) Edit: The authors acknowledge this
in the "Limitations".

------
fsh
I quite like the Spyder approach: Pure python code that is segmented into
cells by inserting a special comment line.

The cells can then be individually executed in an ipython shell, or the entire
script can be run with the regular python interpreter. This makes it easy to
tweak the individual parts without having to re-run everything. In contrast to
jupyter notebooks you still end up with a valid python script that can be
easily version controlled.

I just wish that I could use vim instead of the Spyder IDE.

------
eanzenberg
Notebooks are sort of like democracy, its the worst form of government except
all the others.

You need to pick the best tool for the job, and often times in machine
learning that tool is a notebook.

------
dkleissas
At Gigantum, we're trying to solve some of these issues too. A Gigantum
Project lets you run Jupyter or RStudio in a container that is managed for
you. Everything is automatically versioned so you can sort out exactly what
was run, by who, and when.

[https://gigantum.com](https://gigantum.com)

------
Evidlo
I wrote a plugin for ipython that some people might find useful:
[https://github.com/uiuc-sine/ipython-cells](https://github.com/uiuc-
sine/ipython-cells)

It lets you do linear execution of blocks like in Jupyter, but in a normal .py
file. Obviously more lightweight than Jupyter and you get to use your regular
editor.

------
sunaden
I work at [https://www.deepnote.com/](https://www.deepnote.com/), we are
trying to tackle some of the pains mentioned in the article (setup,
collaboration, IDE features like auto-complete or linting).

We are still early access, but if you are interested in an invite just let me
know. My email is filip at deepnote dot com.

~~~
serendipityisme
Deepnote seems quite interesting, but as a cheapskate grad student, I'm
compelled to ask.

If this information isn't private, what sort of business model do you use? I
take it you'll have a SaaS subscription model? I see it's free to use now, but
how does your company plan to make money (especially taking into account the
cost of the cloud hosting Deepnote requires)?

~~~
sunaden
Hey, thanks for the question.

Our goal right now is to build the most amazing data science notebook. We need
a lot of feedback to get there, that's why we are keeping it free. But since
the servers also cost us something, we haven't opened up Deepnote to the
public just yet.

Once in GA, we know we can support students on a free tier almost indefinitely
(it doesn't really cost that much) while offering more advanced features on a
subscription model for teams and enterprises.

------
kriro
I'm curious. How do people protocol their experiments? When I started, I used
to just keep the cells but that lead to very long and impossible to parse
Jupyter notebooks. I have since opted for keeping a journal.txt file in Atom
where I write down hpyerparameter configurations, epochs run and results (for
ML). But that feels a bit awkward as well.

------
thiagomgd
Small note: why post an image with the pain points if I need to check the list
below to understand what's written?

------
Rainymood
People love jupyter notebooks for the same reason people love Excel.

This is intended to be a Zen-like Koan, so take it as you will.

------
tdhttt
Previous discussion:

General:
[https://news.ycombinator.com/item?id=18336202](https://news.ycombinator.com/item?id=18336202)

Version Control:
[https://news.ycombinator.com/item?id=21661013](https://news.ycombinator.com/item?id=21661013)

------
zneveu
One idea for a pain point not mentioned: better variable persistence. If I
declare a variable, then delete the cell I declared it in, the variable
persists. I've had this cause issues because if I use the deleted variable by
accident, it will work fine right up until a kernel restart.

------
procrastinatus
I’m surprised no one has mentioned what I see as the biggest failings of
notebooks: poor handling of connection loss / re-connection. The kernel will
continue to run, but a connection hiccup will often make the notebook UI stop
updating (and lose any kernel output).

------
commandlinefan
I don’t get the popularity of these things - I think you have to have started
out with them to like them.

~~~
bloaf
I learned to like them in Mathematica, in many ways Jupyter is just a pale
imitation of the excellent system built at Wolfram.

------
ageofwant
For emacs org-mode users this [https://github.com/dzop/emacs-
jupyter/blob/master/README.org](https://github.com/dzop/emacs-
jupyter/blob/master/README.org) is worth looking into.

------
tastyminerals
Notebooks are bad and unreliable. You are repeating your code all the time,
you are limited to work with smaller datasets. If you are into visual data
analysis use Orange or other similar data mining tools. We allow usage of
notebooks only for presentation purposes.

------
cwyers
It is interesting to me how this talks about "computational notebooks" but it
seems to be about Jupyter and derivatives thereof -- RMarkdown notebooks run
inside of the RStudio IDE, and they don't use the term 'kernels' like Jupyter
does.

------
fulafel
What are the currently available CI options for notebooks? You'd think this
would be one of the first tools people would need to make sure notebooks are
reproducible, but there seems to be little sign of CI usage.

~~~
amirathi
Checkout treon[1], open source testing framework for Jupyter notebooks. Since
it runs via CLI you can hook it up to any CI platform of your choice.

Disclaimer: I wrote large part of treon.

[1] [https://github.com/reviewNB/treon](https://github.com/reviewNB/treon)

------
kdamica
Streamlit is imo the best alternative. I was a beta tester and I found that it
encouraged good coding practice without sacrificing too much functionality. I
highly recommend that other data scientists check it out.

Streamlit.io

------
wrnr
Trying my best to solve some of these:
[https://github.com/wrnrlr/foxtrot](https://github.com/wrnrlr/foxtrot)

------
voldacar
Performance is another pain point, at least for jupyter

------
juskrey
Why no Mathematica?

~~~
enriquto
It's not even free software.

~~~
pjmlp
Developers need to pay bills somehow.

~~~
kragen
"Free software" is about freedom, not price. Putting the research that is your
life's work inside of proprietary software that can be taken away from you,
forever, at any time — that seems foolish.

~~~
pjmlp
Copyleft derived "free software" can also be taken away from me.

~~~
kragen
Only if you violate the license.

By the way, I'd be interested in your thoughts on
[https://news.ycombinator.com/item?id=22083468](https://news.ycombinator.com/item?id=22083468).

------
jupp0r
Why not teach data scientists how to write software effectively? Those are
smart people, it’s not like using version control, writing unit tests and
extracting common code into libraries is rocket science.

------
ngcc_hk
What is wrong with life? Many but let us appreciate how to use it more instead
of seemingly criticise it.

The world is so much better with you alive. So is the founded tool of
computational notebook. Not sure I read it covered R notebook which is really
good to share info and analyst. Just wonder how to use it better.

Of course they can always improve on it. But I would promote more expansion -
How about a lisp notebook, a clojure notebook, a js notebook and a forth
notebook.

The real problem is can you have oo notebook ... it is more “serial” and
graphic and data. But not for the “messy” class or trigger Based system. Hence
if I may, the real problem is the scoping. It is so hard to visualise a live
oo system. Unlike a live functional or even a stack based system.

It is not life that is the problem. Even useless life has its use, as long as
it is alive. But if it is not reaching there an alternative may have to think
about. Just like we cannot be there we send in our voyagers outside solar
system.

Be long and prosper.

