
The Future of Notebooks: Lessons from JupyterCon - wcrichton
http://willcrichton.net/notes/lessons-from-jupytercon/
======
stevesimmons
I don't get the people here saying don't use Jupyter notebooks, or they are
bad software engineering.

So much Python development is trying snippets of code in a REPL as you
introspect live objects, then once they're right pasting them into the IDE.

All my Jupyter notebooks are like that, where my code starts as cells of a
line or two, as I check each output. Then I coalesce them into a function
(which avoid the problem of execution order that some people here mention).
Then I may move those functions into a normal Python module and build up
complex classes, add unit tests. Or if the goal is communicating with other
people the results of running that code, I'll add explanatory text and
mathematical formulas, break it up into sections, include references to papers
and web links.

I think what people miss here is many software development tasks, especially
in data science and machine learning, are essentially exploratory data
analysis. And the outputs are graphs, tables of data etc, designed to be read
by humans. Jupyter's ease of use for iteration, and the ability to interleave
documentation, formulas, graphs and tables with the code is a big win.

In my daily work, I have my IDE and Jupyter open side by side. The Jupyter
notebooks are version controlled like my .py modules.

Over time, I expect editors like VSCode to edit Jupyter notebooks natively.
And also more code editing to move from IDEs into the Jupyter (via integrating
the Monaco editor). We will all benefit from that crosspollination, whether we
are principally software engineers or data scientists.

~~~
seanmcdirmid
> And also more code editing to move from IDEs into the Jupyter

Jupyter, and all REPLS in general, rely on a concrete top-level execution
context to provide useful feeedback. Much of the code written outside of data
science doesn’t really have that, which is why you don’t see notebooks being
used for software development very often.

~~~
vosper
I frequently use IPython or REPL.it for software development. It's not my main
editor, and I don't write large chunks of my code there, but for rapidly
prototyping a small subset of some task I find them invaluable.

But you are right about notebooks - I don't find them particularly useful,
except as a better REPL, and no-one I know uses them for "normal" (non Data
Science) software development.

~~~
AdamM12
I think my favorite non "data science" use case is web scraping. Ad-hoc nature
of the task lends itself to using a notebook. A lot of times I don't want the
full suite and boilerplate Scrapy gives you so I write it out in a notebook
with Parsel & Requests. Once done export and clean up.

------
jakelarkin
I understand why they became popular, but as a software engineer considering
how they work, I am just full of disappointment. we're going to spend the next
ten years re-inventing every single software engineering best practice for
jupyter's weirdo environment.

~~~
zmmmmm
This is definitely one of my concerns too. Ad hoc code inside these notebooks
is almost completely unmanageable from any reasonable software maintenance
perspective, and refactoring code out of them is prohibitively difficult as
well. I really want something to emerge that combines the best of both worlds
of an IDE and notebook development, but there isn't anything close currently.

~~~
staticassertion
What? What are you using Jupyter notebooks for where you want maintenance?
They should be records of data analysis/ procedures, not code that runs in
production or something.

~~~
laichzeit0
Maybe read the article. They are experimenting with putting the notebooks
directly into production ala bash script. I don’t think this is a great idea
either.

~~~
staticassertion
Yeah, I read it after, silly of me to comment first.

------
pyrce
As the speaker for the scheduling notebook talk that got referenced here, I
can probably give a few insights about how we're using notebooks. A lot of the
risks and concerns brought up here were talked about in the session. The
slides ended up on [https://conferences.oreilly.com/jupyter/jup-
ny/public/schedu...](https://conferences.oreilly.com/jupyter/jup-
ny/public/schedule/detail/68348) (and hopefully the talks themselves will get
posted soon).

In particular I referenced how we treat and emphasize notebooks as an
integration tool which acts as a good place to combine actions with
documentation, visuals, and output logs. There's a section on Integrating
Notebook which outlines how we approach this. There's also a strong emphasis
on pushing complexity and shared code into the repositories housing notebooks.
Effectively you end up with a lot of same best-practices found in non-notebook
development, and the same abuses that lead to unmaintainable code -- which
sometimes is needed in the short term.

So far we've had a lot of success with notebooks in production as
parameterizable templates, or as a way to easily produce scheduled reports or
machine learning experiments. Many users just provide the parameters while
supporting teams provide the tested templates. Other users like being able to
simply schedule their iterated work without needing to translate to another
medium. One of the biggest wins though is gaining a shared interface for
debugging, experimenting, and reusing code by having notebooks as the output
artifacts of execution (even if it's just executing some other code elsewhere
on behalf of the user). Papermill made a lot of this possible by separating
input notebooks from output notebooks, and by being able to inject runtime
values into those output notebooks.

~~~
pyrce
We also are doing a blog series on how we're approaching notebooks at
[https://medium.com/netflix-techblog/notebook-
innovation-591e...](https://medium.com/netflix-techblog/notebook-
innovation-591ee3221233) (the scheduling post is the second in the series).
Also I'd recommend listening to talks from the conference with an open mind.
There were a lot of great discussions and positive energy around notebooks.

Hopefully these help describe a few of the new patterns and tools available in
the notebook space.

------
xvilka
It is a disaster that more than half of the research papers do not provide
data and the code, required either to reproduce research (if it is a
theoretical one) or reproduce results analysis. Wider adoption of JupyterLab
(and alike) will help to solve this problem. Using PDF and paper for
distributing research results feels like XIX century these days.

------
aportnoy
Most comments here express disbelief and disappointment in Jupyter from the
software engineering point of view.

What exactly is wrong with it? I use Jupyter daily and find no other Python
environment more productive, be it scripts or IPython or IDEs. Granted, I work
in scientific computing and use Python for data wrangling and stats. I find
immense value in interactivity and iteration speed.

~~~
dnbgfher
Maybe I can provide a bit of perspective on this, as I have lots of
conflicting feelings about Jupyter.

When I'm doing some bit of data wrangling or just exploratory work with data I
quite like it - at least at first. As you pointed out, it's really easy to
extremely quickly iterate on things and start to get an idea of what's in the
data, what techniques work, and which don't. It's great.

Until it isn't. I'm probably "using it wrong" but all the suggestions I've
seen for doing it "right" start cutting into the iteration time. If I'm in
Jupyter it's because I want to flail around real fast and see what happens.
And that causes all sorts of problems. Which cells do I need to rerun
together? Which cells should I _not_ run again. What the heck is actually in
all these variables right now anyway, because I don't remember which order
I've run (and rerun) all the cells in. And what was that one approach or
parameter that really worked well that one time? I don't remember.

These sorts of things aren't an issue with a non-Jupyter approach. And because
I'm taking my time anyway I'm probably being diligent with source control, and
all that. Even pulling code out of a notebook into a more stable workflow is a
pain, because you have no assurances you can just copy/paste a cell and have
it work. In my experience, it never will.

It's one of those things that has a pretty heavy costs and benefits. And
unfortunately they aren't really possible to separate.

Not coincidentally, I feel the same about dynamic languages and even less
usefully typed static languages. Jupyter takes an already fairly extreme
position on the stable(safe)/easy continuum and really ramps it up to a point
where it's hard to wrangle time and work done easy into some sort of
stability. It's great, and it's awful.

~~~
moultano
I think notebooks could be hugely improved by saving a snapshot of state after
each cell, and having the linear order of cells only move forward in time.
Accidental variable reuse (especially from a cell that is in the "future" or
maybe no longer exists) is a huge source of bugs.

~~~
dnbgfher
From other comments, it sounds like something called Cocalc might be of
interest to you. I haven't even looked at it yet but it sounds like it does at
least some of those things.

------
drej
People complaining that notebooks are not good for software engineering are
missing one important point – notebooks are very often used by people who are
not software engineers and/or not doing software engineering work.

(The point is otherwise valid, I have seen software engineering in notebooks
and it was horrific.)

------
jdonaldson
"jupyter is the new bash"

As much as I love Jupyter this is a bridge too far even for me. There's
nothing about Jupyter that makes sense for non-interactive scripting.
Although, I suppose it would help you do block chain calculations... which
would also be a terrible idea.

~~~
adw
Inline documentation and charting (which for data tasks is a honking huge
deal). It's literate programming with an actual use-case. Very into this idea.

~~~
seanmcdirmid
None of that is what people use bash for, however.

~~~
adw
You'd be surprised. It's certainly not _all_ of what people use bash for, but
there are a lot of sketchy ETL (and sampling, and small-scale analytics) jobs
out there held together with bash and awk and hopes and dreams, and replacing
those with something like this is a substantial win.

------
celias
If Jupyter is not your cup of tea you might like Cauldron, the unnotebook -
[http://www.unnotebook.com](http://www.unnotebook.com)

Interview with the author in May, 2017 - [https://www.linkedin.com/pulse/why-
cauldron-might-right-data...](https://www.linkedin.com/pulse/why-cauldron-
might-right-data-analysis-environment-you-dan-mayhew)

------
abetlen
As a pretty heavy notebook user (for computer vision and machine learning), I
have to say that all of these new features are nice but they don't address the
real problems with the current ecosystem. Namely, there's no good way to
transition exploratory notebook code to python modules. We need linting and
refactoring tools _in_ the notebook and something better than `aimport` for
moving code into separate `.py` files.

------
eanzenberg
I despise notebooks being used in production for the reasons given early in
the article: you can execute code in any order so authors usually end up with
spaghetti and long pages without clear flows. I would rather scientists use
and learn the tools that have been developed over decades on collaborative
code writing using version control. It helps integrate their solutions too.

~~~
logicchains
This is a code style problem not a notebook problem: the same people would
probably write spaghetti without clear control flow if writing a Python batch
script instead of a Notebook.

~~~
eanzenberg
I agree, but tools like version control do help a ton to manage and mitigate
these kinds of risks. Actually, it could be super super cool if someone
figured out how to version control notebooks, including tracking of code
execution flow.

------
michihuber
To everyone frustrated with notebooks, I urge you to check out
nextjournal.com:

\- Notebooks are automatically versioned

\- You can reference and reuse parts of other notebooks immutably through
something they call "Transclusions":
[https://nextjournal.com/nextjournal/transclusions](https://nextjournal.com/nextjournal/transclusions)

\- The technology-stacks underlying each notebook are immutable, meaning that
notebooks work on any machine (no "hidden" dependencies).

\- You can explicitly reference other results instead of relying on global
state, making execution order irrelevant.

It's currently in private beta, but people are starting to use it in
production. (EDIT: You can sign up using the code curryon2018)

(Disclaimer: friends of mine are building this and I used to work on it in the
past.)

------
enriquto
I love jupyter notebooks, but unfortunately the code editor is unusable to me.
I cannot stand the bizarre parenthesis auto-completion, and the
autoindentation settings; and there's no easy way to remove them.

~~~
Tistel
I feel the same way. One workaround is that some editors have a plugin. Emacs
has a mode called ein that is fantastic. As you poke around getting the
snippets to work, you can cut/paste into a different file/buffer. After you
start the Jupiter notebook copy the login token, run: ein:notebooklist-login
then ein:notebooklist-open and away you go.

~~~
mi_lk
Just be careful to use [https://github.com/millejoh/emacs-ipython-
notebook](https://github.com/millejoh/emacs-ipython-notebook) instead of the
unmaintained [https://github.com/tkf/emacs-ipython-
notebook](https://github.com/tkf/emacs-ipython-notebook). Somehow the latter
comes up as my first search result.

------
ur-whale
"On the education side of things, Jupyter is quickly gaining adoption in
universities around America, particularly for data science courses.
Conversely, data science is increasingly becoming students’ first exposure to
programming and computer science"

Am I the only one this two sentences worries deeply?

Or is it just a rehash of the twenty year old "my first exposure to
programming was Excel" (which, from what I'm reading is what the article is
talking about : a glorified spreadsheet).

------
dangjc
I can’t recommend enough using Atom with the Hydrogen plugin. A Jupyter kernel
runs in the back and you can execute a line or multiple lines of code at once
just like the web notebooks. But you’re editing a plain code file, not an
.ipynb, which is easy to check into git, move to inside a library, and get
full IDE tooling on.

------
gaius
When you consider that it is impossible to reuse notebooks in each other... Or
unit test them... Or that it is not especially easy to version control them in
any sort of branch/merge workflow... Jupyter Notebooks are much closer to
Excel spreadsheets than they are to what most people would consider actual
programs.

~~~
logicchains
>impossible to reuse notebooks in each other... Or unit test them...

It's quite easy to unit test notebooks. Whenever you write a function you want
to test, write some unit tests (or even just asserts) in the same cell it's
defined, then any time the cell is run to define the function, the tests are
run too. Likely you'll be doing some manual tests when you write a block of
code cum function, and it's not hard to copy the input crafted from those
manual tests to a unit test.

~~~
gaius
_write some unit tests (or even just asserts) in the same cell it 's defined_

As many people will do this as presently write tests for their Excel macros or
formulas.

------
aluren
It would seem that notebooks were specifically tailored for my use cases
(being in academia and doing data analysis), in practice I found that I drift
away from notebooks every time I try them and stick to a good old console
(ipython or bash) instead. Here are a few things that irked me:

-It's browser based. Yuck. This means you can wait upwards of ten seconds for a session to load up every time you want to start one. Also, generally speaking, I deeply dislike browser-based applications that have nothing to do with the internet. It just feels so clunky. My workflow gets interrupted as I have to play between windows _and_ betwen tabs, and whenever I want to switch from a task to another I have to remind myself 'well this is my web browsing instance that also happens to be my notebook', it's not natural and gets in the way. By contrast, a terminal advertises itself as exactly what you want to use it for. It's also very snappy.

-Following up on browser-based awkwardness, my god is this thing slow. Initializing kernels. Interrupting kernels. Restarting kernels. God save you if you want to import more than a couple libraries. Wait you made a mistake and want to stop and restart that one cell? Guess you just have to go through the whole kernel thing again I guess. So you launched something and it turns out it eats way more memory than expected? Well, 'sudo pkill -9 firefox' it is, do not pass go, do not collect autosave. It just does not play nice and lags behind every thought or action you want to take. Again, contrast with a plain terminal that just does the job and does it _now_.

-Speaking of exiting, these things don't seem to be able to do so gracefully. If something hangs up you just have to kill everything if you want to resume working, instead of good old ctrl-C. The fact that killing everything _also_ tends to kill the web browser you used to do actual web browsing is just icing on the cake.

-Setting things up can be dodgy. Should your notebook fail to start, error messages are extremely cryptic. Documentation for troubleshooting is lackluster. No one ever seems to have any two identical issues on Stackoverflow, which is also plagued with WorksOnMyMachine (tm) syndrome. If you ever have to mess up with both Python 2 and 3, you may be in for a good time.

-The notebook format in which sessions are saved is supposed to be human readable and editable. In practice I have never seen a single human read or edit any of these by hand, for the very simple reason they are horrible to read or edit by hand. Again, why do I have to load a file in a cryptic format in my web browser when I could just %paste my snippets off of my editor or even just use my IDE's builtin capacities (code runner on vscode, F9 in spyder, etc.)? Plus, compatibility issues between versions can make your notebooks suddenly unusable (and hard to salvage due to the aforementioned format issue). This doesn't happen with snippets pasted onto a terminal.

-IDE integration has been sloppy in my experience. I've been told this may be subject to change, however.

Note that all this criticism doesn't mean I don't respect the work of the
people behind this - they've made a fantastic tool that just doesn't happens
to fit my use cases, for now. Every once in a while I try to work with
notebooks, find they're still unusuable, shrug and go back to my terminals. As
the experience improves I may make the switch one day, who knows.

~~~
alper111
I agree that probably working on bash feels more snappy but Jupyter is not
that clunky as you mentioned. My laptop is a mediocre one and I can run the
Jupyter, navigate to the code and open the editor in total like 3 seconds.

In my experience, kernels work just fine unless you are doing something you
shouldn't. My kernel problems occur at points where I have a memory issues.
Though I agree with you that you don't understand what's wrong once the kernel
dies.

~~~
aluren
>In my experience, kernels work just fine unless you are doing something you
shouldn't.

Here's the thing. I'm sure many of the complaints I voiced can be chalked up
to lack of experience. But even if it weren't for the issues about
documentation and troubleshooting, I'd still have yet another tool to manage,
learn to use, learn best practices, etc. Instead of just getting things done
with my tool it just gets in the way and tells me there are things I should
and shouldn't do. It takes time to learn about it.

And now, I'm not sure the investment is worth the effort. Despite its warts,
everyone uses bash and it's really easy to learn about it and translate that
knowledge into many applications. And, it _does the job_ when I ask it to.
Same for Python, IDE/editor usage, LaTeX, and all the other toys in your
typical academia toolbox. I'm all for learning. But time is short, deadlines
loom close, and the good enough is often the better's worst enemy. And when it
turns out the better isn't actually better, welp.

------
ausjke
Been a bpython lover now I mainly use ipython. Used jupyter + ipython kernel
once a while but for 99% my use cases, ipython is good enough(testing out
code, checking out help info,etc).

For anything you want to have graph(e.g. matlab), notebook shall serve you
well.

------
Eridrus
I like notebooks. I like them for data work. I like them for prototyping
things that many may consider software engineering. I still think this
presentation has a lot of good points about where they could improve.

------
megamindbrian2
I wrote an importers for notebooks that laods and runs the cell based on
keywords I'm the markdown.

------
plg
from Joel Grus, “I don’t like Notebooks”

[https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUh...](https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUhkKGvjtV-
dkAIsUXP-AL4ffI/)

------
gumby
I found this most exciting:

> While Jupyter notebooks have traditionally been a humans-only entrypoint
> into a program, researchers and companies alike are increasingly using
> notebooks for automation.

It's part of a slow journey back the power of the Lispms and the Smalltalk
environments.

------
agumonkey
"jupyter is the new bash" has to be the most underwhelming hyperbole of the
decade

~~~
tree_of_item
Are you kidding? The shell is a massively important part of modern computers,
and making it more expressive would be a huge win. You might be familiar with
TermKit[0] which also tried this.

[0]: [https://github.com/unconed/TermKit](https://github.com/unconed/TermKit)

~~~
agumonkey
${SHELL} is\ horrible\ and\ dated

------
h4b4n3r0
Even in its present state, Jupyter is strictly better than plain REPL,
especially if you need visualization. It's like Python in general: it was not
designed to write 10-100KLOC programs. Python was designed for scripting.
Jupyter was designed for interactive data exploration, and to create a record
of results which you can view without re-executing the cells. Is it error
prone? Yes. But then so is REPL.

~~~
stevesimmons
> Python was designed for scripting.

It's far more than scripting, and for much bigger programs than 10k LOC... My
company's Python codebase is 35m LOC/500k modules, with 25k commits per week
and contributions from 2.5k developers per month.

To my mind, Python has the opposite problem. I've been a Python programmer
since 2000. Every few years, I think I should devote more time to other
languages (first Java, then R, then Haskell, then JS, ...). But the Python
ecosystem just keeps getting stronger. My current focus is data science and
machine learning. In these domains, there are no good reasons to drop Python.

~~~
h4b4n3r0
JS was also designed for scripting

------
keeptrying
You should never use notebooks - period.

My TL;DR: • Confusing for beginners • Encourage bad practices • Poor editor

~~~
aportnoy
Why not?

~~~
keeptrying
My TL;DR: • Confusing for beginners • Encourage bad practices • Poor editor

------
grillorafael
As many already said, I think it is a great tool for prototyping and data
exploration but when it comes to moving code to production, for me it makes
very little sense to use it.

Netflix said that if the job breaks they can enter the notebook with the data
and see what is wrong. For me it feels like they did development with 0 safe
guards and if it breaks they check why. Instead of logging problems and
dealing with edge cases in the code beforehand

~~~
jre
I'm not sure exactly what Netflix means by "if the job breaks", but I've used
jupyter notebook in production to do ML before.

If you consider the notebook as a way to augment your logs with plots, it
might make a bit more sense.

Running a jupyter notebook is a nice way to generate a HTML report for a job.
For a typical ML pipeline, you first plot some stats about the input data,
then train some model, plot some training loss, a confusion matrix, some
example of predictions, etc...

If some job gives a strange result (maybe that's what they mean by break),
having the notebook rendered as an HTML page with all the plot is a very
effective way to do a first round of diagnostics. You can also start the
notebook with the same parameters and 'run' through your report, which is a
nice way to do interactive debugging.

Also in this case, the notebook itself was quite small in terms of lines of
code. All the functions were implemented in modules, so it's really like the
notebook is your 20 lines 'main' function. So you need some discipline among
your team.

