So much Python development is trying snippets of code in a REPL as you introspect live objects, then once they're right pasting them into the IDE.
All my Jupyter notebooks are like that, where my code starts as cells of a line or two, as I check each output. Then I coalesce them into a function (which avoid the problem of execution order that some people here mention). Then I may move those functions into a normal Python module and build up complex classes, add unit tests. Or if the goal is communicating with other people the results of running that code, I'll add explanatory text and mathematical formulas, break it up into sections, include references to papers and web links.
I think what people miss here is many software development tasks, especially in data science and machine learning, are essentially exploratory data analysis. And the outputs are graphs, tables of data etc, designed to be read by humans. Jupyter's ease of use for iteration, and the ability to interleave documentation, formulas, graphs and tables with the code is a big win.
In my daily work, I have my IDE and Jupyter open side by side. The Jupyter notebooks are version controlled like my .py modules.
Over time, I expect editors like VSCode to edit Jupyter notebooks natively. And also more code editing to move from IDEs into the Jupyter (via integrating the Monaco editor). We will all benefit from that crosspollination, whether we are principally software engineers or data scientists.
As soon as I have one piece of functionality working as a function, I move the code to the "main" file for the project and import the function into the notebook. This means I never have to write more than a few paragraphs of code in the notebook and instead use it for testing and glue code. I don't think of the notebook as something sharable with others, it's more like WIP code... if I keep the notebook around it's usually because it can end up a useful test harness when code needs to be revisited/updated.
As for global state, one thing that I find really helpful is the keyboard shortcut 00. If you focus anywhere outside of a cell and you press zero twice, this restarts the kernel and you can re-run the commands from the beginning in a "clean state" so what you see matches what is. This approach works well when you're editing the source code that gets imported into the notebook---you never have to worry about re-importing since always restarting from scratch.
You have scientists. They are trying new things, and while reproducibility is nice, most of the time it doesn't matter. Tests don't matter, to put it mildly, because we're at a stage where it isn't just "not a product", it's not worth to become product, and they are aware of this. IF there is an end product, it's a report. A report that lists in general terms what was done and what the result was, nothing more.
Their goal is to fully understand the mechanics, the minutiae and being able to explore and tinker with it (without tests breaking) is valuable. I hope you can see that most tests get very much in the way of this activity. I'm not saying there isn't anything you might test (quite the contrary), but you're not going for 100% coverage. Hell, probably not even 10%.
That's how you get to matlab, mathematica, python notebooks, ... That's the usecase.
On the other hand you have software engineers. They're also not a homogeneous group, but let's ignore that. The key point is that they do not understand the details and don't intend to understand the details. The goal is also different. The goal is not to learn, the goal is to help a company build a product. Reliable, repeatable, controllable, and something where you can pick someone of the street and tell them to fix something and actually expect that to get fixed.
Understanding everything about everything you work with, like an academic would try to do, is not practical for a software engineer.
And they get presented with an excel sheet, a notebook. They don't understand, there's no docs, but remember, they don't want to learn. Worse ... likely there's problems with the notebook. Maybe it solves one case particularly badly for some reason. Maybe it needs to run in an entirely different environment, like maybe on a phone. Or faster. Or needs to start doing something new.
So they need to change something. Quickly, and quickly means without understanding the domain behind the code. So they change something and ... everything else falls down.
So software developers want properly designed code with comments about even banal things, explanations about every last variable, with more tests than code, that cannot reasonably be changed with less than 5 people, and where you can always and without thinking get it back to when "it worked". Without understanding the domain behind it.
So notebooks are not for software developers.
You'll see this difference reflected in the languages as well. Notebooks are for languages that allow one to talk extremely high level, extremely succinctly. Because that's what academics want to do.
Software developer languages, the successful ones, are basic simple things, like (these days) Java, C and Go (even C++ and Python are considered too high level by most and the frameworks software devs work with don't exactly have PyTorch's flexibility). If you want to see how you give software devs what they want, study Ada (and resist the urge to gauge your own eyes out afterwards). But there is no language, imho, as strongly on the software developer side as Ada. I would even say it even throws basic sanity under the bus to get controlibility.
I would say to some extent machine learning and software development are enemies. You can fix a number of things, but when push comes to shove, machine learning is about "just figure it out", which is fundamentally against the software developer mindset of controlibility. Can you guarantee that a NLP model will never be racist ? The truth is ... not really. You can test specific cases, but the whole point of using such systems is creativity.
Use the right tool for the job. Although I would say, to learn programming ... even for software devs notebooks are great for this.
> So notebooks are not for software developers.
You admit software developers are not a homogeneous group, but these are fascinating blanket statements none-the-less. "Notebook" systems originated from software developers, if you allow the origin to be Knuth's ideas of Literate Programming  as the forefather of Notebook languages.
It seems an interesting full circle of sorts that Knuth heavily promoted the idea of Notebooks as a superior programming environment in a time when the languages weren't particularly suited to the task (compiled languages that needed very delicate surgery to convert from human readable order to compilable code) and tools/environments that were effectively suboptimal to the task (text editors with no GUI/WYSIWYG abilities; macro-based compilers instead of live interpreters; etc). Now we finally have the tools to make that work, and at least in a few places developers are starting to make use of it (Netflix's Papermill in the linked article is an interesting example).
Admittedly, you might argue that Knuth was perhaps more on the academic side of the fence than software developers today, but Knuth very succinctly argued that all software development would perhaps be better if we stopped relegating the human/natural language narrative of the project to embedded comments in a programming language, and instead embedded the programming language into our human/natural language narratives.
Alternatively one can write exploratory code as unit tests and run them inside an IDE. When the exploration is done, three quarters of the tedious cleanup effort is already done. There are already unit tests. There are already APIs exercised by the unit tests. The code is in much better shape.
The missing two bits:
* No charting support. To the best of my knowledge, neither PyCharm nor VSCode have an output console with HTML support.
* Caching. Some operations take a lot of time. At the extreme, fast.ai style development, where some cells train a model for hours/days. Re-running such code from scratch on every test run is impractical. An opportunity for a library with explicit save/load/cache/checkpoint support.
I made a really simple python cli module that would convert from jupyter notebook to python files with runnable cells in vscode, and vice versa.
Really upped my game while doing the udacity courses.
Vscode does indeed have a html output console.
> Alternatively one can write exploratory code as unit tests and run them inside an IDE. When the exploration is done, three quarters of the tedious cleanup effort is already done. There are already unit tests. There are already APIs exercised by the unit tests. The code is in much better shape.
Jupyter, and all REPLS in general, rely on a concrete top-level execution context to provide useful feeedback. Much of the code written outside of data science doesn’t really have that, which is why you don’t see notebooks being used for software development very often.
But you are right about notebooks - I don't find them particularly useful, except as a better REPL, and no-one I know uses them for "normal" (non Data Science) software development.
Please, please, please. Given the rising importance of interactive programming, I really think it deserves more than a extension to give us a seamless experience.
I have been having a hard time getting comfortable with that idea. There is usually "a lot of" untested and unreviewed code in the notebook that produces the analysis.
- Code is small, most of it is calling well-know libraries with the algorithms.
- Every 3-4 lines of code you usually show the results of what is happening (either how the data changed, a graph or whatever)
- 95% of the errors in data science come from bad practices: you either didn't clean up the data correctly, you didn't split the train/test/validation sets correctly or at the right step in the process, you chosed the wrong algorithm, etc
- most of the time you don't know whether you are doing something wrong or not because you don't have enough knowledge, but the code shows something all the time. No bugs in the code.
Those are the reason I don't care very much about bugs in the code because usually there are bigger problems in the data science process rather than in the code.
PyCharm has support for Jupyter notebooks , but for now it's not very good . Two weeks ago Jetbrains released this official comment :
we are aware of the problems our Jupyter support has and we aren't going to fix them, because the current support is going to retire.
Currently we are working on another support of Jupyter notebook (even 2!), which hopefully will be better and much more usable.
We plan to make it ready for 2018.3 PyCharm release, so stay tuned.
My note: The current PyCharm is version 2.2.
I have a python window open on my desk right now. But the model of programming here contrasts sharply, in my mind, with the one discussed yesterday, from HtDP https://news.ycombinator.com/item?id=17826959 .
I absolutely use the REPL, for instance to determine where to put what arguments into a library function. But I have also in the past had it happen that I found a routine of mine that missed an edge case and there were a good number of projects that subsequently used that routine, that now could be wrong. Some of those could have led to a publication, so there could in that sense be a permanance to that error. Oops.
Those of us who have gotten bit, or who teach and routinely see students develop habits we know will lead to them getting bit, worry about systems that are not developed so much as pasted together.
That's interesting, could you give us some more details ? Last time I tried to put a Jupyter notebook under git it was as mess. Do you use another tool than git or have they made tools to help with version control ? Or is it just your workflow that helps, like emptying all cell results before saving ?
For Jupyter notebooks, cell results get emptied before saving, unless declared public. That is to ensure data confidentiality, not really to help version control.
Of course, standard diff gets confused with ipynbs. You need a tool like nbdime (notebook diff and merge).
With that approach, though notebooks are clean they're still fairly poor for easily evaluating diffs between versions. If code review / diffs are more important than preserving the notebook, then you could use a post save hook to convert notebook input to a .py file and output to .html:
How many businesses rely on bodged up Excel spreadsheets to run their business?
This is that, but on steroids.
For better or worse.
Then: `pytest -s play.py`
I just don’t understand what the appeal of Jupyter is, but iPython is great.
This approach to development might be a lot rarer than you think – I've never seen anyone do it frequently. For tiniest snippets ("If I want to split there and then use that suffix--how do I need to index? Let's check in the REPL real quick" or "What did that stdlib function return again? [Because that's of course not documented, why would anyone need to know that, after all]" and stuff like "I have this list of tuples like that, now, how do I need to zip/unpack/star that stuff such that I get a list of X instead. Let's try in the REPL with [(1,2), (3,4)]!"), sure, for "actual code" that's going somewhere... nope.
But perhaps the biggest strength is that you go through Pandoc, so you can just click a button and get the output as a Word file for sending to someone who's not a software dev, or you can get it as a LaTeX source file and extend it into a proper formal document like a journal paper, or you can do it as HTML where embedding Bokeh scripts and other interactive things Just Works.
I've even made presentations with it going through the reveal.js framework, where you can put an interactive plot on one of your slides, and show people live "what happens when we change this parameter". That's still semi-witchcraft in 2018, but it's going to become a common thing (hopefully).
- easily manage multiple kernels (in different languages / machines) in one file
- tree-based organization manages complexity better than linear notebooks
- no browser in sight (unless you need interactive widgets)
- highly exportable, including to ipynb via ox-ipynb
- small user base / limited docs
- can't easily import from ipynb
- async cell execution support is early-stage
See Scimax ipython for examples
I’m also much more excited by R Notebooks as implemented by RStudio than I am Jupyter. R Notebooks take the same basic approach as org-mode, implementing a smaller set of functionality built around a more mainstream-palatable Markdown format.
I don’t like R Notebooks as much as org-mode (why reinvent the wheel!?), but at least the general approach plays nicely with git.
On the other hand, the lisp-addled part of me starts to think that the juypyter format being json might actually be a step up from a poorly specified markup format that has to be parsed into data structures.
Maybe the real issue is that git is an insufficient revision control system, and that we need revision control systems that can revision data structures, rather than simple text diffing.
Usually, new code starts in cells with some Org-managed context (e.g. a Jupyter kernel in a remote container with some DB/service access). This is done using the :session code cell keyword, which works per subtree. Managing remote sessions like this generally keeps me away from terminals.
Surrounding the cell are various mini-dashboards with useful docs / links / commands for that part of the project. Since Org supports embedding elisp and shell commands in clickable links , these mini-dashboards can be made very quickly.
Org lets me edit the code using the proper Emacs mode for its language, while pulling dynamic completion / docs from the Jupyter kernel. Just like Jupyter notebooks, I can view rich outputs from the cells in-line. I can then name the outputs and make them inputs to other cells, including ones in different languages / kernels. AFAIK that's an Org-only trick.
Most code eventually finds it's way to normal source files (see Org's "tangle" feature). This feels more natural than moving code from notebooks since, again, the cell editing mode is the same as the one for source files.
Org's tree-manipulation capabilities + support for multiple sessions means that (so far) I've only ever needed 1 Org file per project. I track this in git, which is simple since Org is just plain-text. To share with non-Org users, I usually export to ipynb  or, for static docs, HTML .
"A Netflix engineer described how they have replaced Bash scripts with Jupyter notebooks for ETL pipelines and cron jobs."
The value of Excel is that is a zero-config tool, available everywhere as 'standard' business installation, allows very quick iteration with visual output in certain range of tasks, is battle tested in millions of computers... etc. And everyone else is using it too.
For a programmer it's easy to suggest to just pick some good language. Once you know two programming languages you can navigate pretty soon fluently in third.
But picking up the first language fluency? That's hard. You can't just suggest to use python or such. You need to provide a tool that holds hands, has lots tutorials available etc.
Visual programming is not necessary the answer either. I've understood that without discipline LabView programs become actual visual spaghetti pretty quickly.
In particular I referenced how we treat and emphasize notebooks as an integration tool which acts as a good place to combine actions with documentation, visuals, and output logs. There's a section on Integrating Notebook which outlines how we approach this. There's also a strong emphasis on pushing complexity and shared code into the repositories housing notebooks. Effectively you end up with a lot of same best-practices found in non-notebook development, and the same abuses that lead to unmaintainable code -- which sometimes is needed in the short term.
So far we've had a lot of success with notebooks in production as parameterizable templates, or as a way to easily produce scheduled reports or machine learning experiments. Many users just provide the parameters while supporting teams provide the tested templates. Other users like being able to simply schedule their iterated work without needing to translate to another medium. One of the biggest wins though is gaining a shared interface for debugging, experimenting, and reusing code by having notebooks as the output artifacts of execution (even if it's just executing some other code elsewhere on behalf of the user). Papermill made a lot of this possible by separating input notebooks from output notebooks, and by being able to inject runtime values into those output notebooks.
Hopefully these help describe a few of the new patterns and tools available in the notebook space.
Even with these benefits I do think the criticisms about software development are right on point. Notebooks are a step backward in terms of software engineering environment with none of the modern tooling, version control, testing frameworks etc. I think that the folks who dismiss notebooks as a platform though are missing some important benefits that have long been absent in current editors. Larger companies like Facebook and Google are already facing the reality that devops is a pain point even for sophisticated software engineers and have developed remote code editors like cider and nucleotide to try and enable bringing code development to the data and compute rather than doing it from the laptop. R has been working on a long time on integrating analysis results and code in a reproducible package with sweave/knitr and python now has pweave in a similar fashion.
I hear all the issues with jupyter and I'm not particularly married to current form of notebooks. I do think though that the features of remote development, data visualization and support for literate/report programming and sharing ode are first class features that I'll continue to want in the future.
What exactly is wrong with it? I use Jupyter daily and find no other Python environment more productive, be it scripts or IPython or IDEs. Granted, I work in scientific computing and use Python for data wrangling and stats. I find immense value in interactivity and iteration speed.
I think out-of-order or unknown-order execution is the biggest source of my own mistakes / bugs. If the order in which your cells are executed affects your result, you've got a bug waiting to bite you or someone else. I've been there. And we tend to run cells more than once, while refining our work.
This potentially affects reproducibility, which is a tenet of scientific computing. So it's important in our world.
I personally think there might be a couple of ways to deal with this issue. One is to put handcuffs on Jupyter and inadvertently make it more difficult or cumbersome to use. But this might be valuable if the size of your project and importance of your result warrants it.
Another way is for us to recognize that at some point we have to grow up and learn some techniques from the programmers. In my revisionist history, software engineering began when programs got too big to see the whole thing on one screen and intuitively understand.
One of my little habits is that I do a "restart kernel and run all cells" when I'm ready to take a break or close out at the end of the day. If if fails, the causes of the problem tend to be superficial, and nothing to lose sleep over.
Another habit is to take cells that work and are nice, and enclose their innards in a function so that their impact on the global variable space is minimized. This can be done during a sleepy time in the afternoon when you're not really in flow anyway.
To avoid the state problem, I start each cell (or, in org-mode, each src block) with a command to set up the environment for that cell, usually by importing a script named something like setup_abc123.py. Cells are not allowed to reference code or results from other cells except through the filesystem, which is under the watchful eye of git. In this workflow, cells exist only to write plots or tables into the notebook for human consumption.
This workflow, for me, has completely eliminated errors due to hidden state dependencies. Before I adopted it I made a serious state-related error every couple of days, which was intolerable.
As a bonus, using a single script to set up the analysis environment makes it trivial to start a new analysis, even with an entirely different tool chain.
Laughing, politely, at that summary of "history." Software engineering started when there were no screens. Code was written on typewriters (called "terminals") and then printed to cards or long rolls of paper (and occasionally using switches to indicate the 1's and 0's you wanted set as bits in your code). Before the Mac a "screen" of code was pretty universally 24 lines of green or white text on a black background, 80 monospace characters per line. A few specialized terminals had 25x80 so they could show a status line below the 24x80 standard area, but they were rare beasts. Yes, there were a few systems like Plato that predated the Mac but if you need a tl;dr it's the observation that, like most revisionist histories, this one is unfortunately total nonsense when looked at in the context of actual history (even if lots of people who didn't experience the actual history believe it).
Before screens, there were pages, before pages there were cards... going back to plugboards, etc.
Each of us got in at a certain time point. When I learned programming, we had a workstation that simulated a card punch, let you see the contents of one card at a time on a little vacuum-fluorescent display, and stored the data on an 8" floppy. You handed the floppy to the computer operator, and received a printout on green-bar paper.
The other way I got into "programming" was staring at a nest of wires, reading my output on an oscilloscope, and making corrections with a soldering iron. ;-)
The common theme was that a program gets too large for somebody to look at and understand it, much less for a team to work on it all at once, unless disciplines are adopted that tend to come from software engineering. When I'm mentoring colleagues who are just beginning to get into scientific programming, some of these simple techniques, such as avoiding globals and creating functions, are in fact an improvement.
When I'm doing some bit of data wrangling or just exploratory work with data I quite like it - at least at first. As you pointed out, it's really easy to extremely quickly iterate on things and start to get an idea of what's in the data, what techniques work, and which don't. It's great.
Until it isn't. I'm probably "using it wrong" but all the suggestions I've seen for doing it "right" start cutting into the iteration time. If I'm in Jupyter it's because I want to flail around real fast and see what happens. And that causes all sorts of problems. Which cells do I need to rerun together? Which cells should I not run again. What the heck is actually in all these variables right now anyway, because I don't remember which order I've run (and rerun) all the cells in. And what was that one approach or parameter that really worked well that one time? I don't remember.
These sorts of things aren't an issue with a non-Jupyter approach. And because I'm taking my time anyway I'm probably being diligent with source control, and all that. Even pulling code out of a notebook into a more stable workflow is a pain, because you have no assurances you can just copy/paste a cell and have it work. In my experience, it never will.
It's one of those things that has a pretty heavy costs and benefits. And unfortunately they aren't really possible to separate.
Not coincidentally, I feel the same about dynamic languages and even less usefully typed static languages. Jupyter takes an already fairly extreme position on the stable(safe)/easy continuum and really ramps it up to a point where it's hard to wrangle time and work done easy into some sort of stability. It's great, and it's awful.
I gave a talk at JupyterCon this week about Cocalc which partly solves this one problem by providing a Time travel slider with complete history.
Right. Get that all the time. My solution is to try to condense the useful code built up across different cells into a reusable block, most often a function. Do that while the 'state' is still fresh in your mind and you remember the order of execution.
It also helps to break up a long notebook into sections using headings and markdown cells with comments. I often start by writing down a question/problem I am trying to solve in prose, that makes it easier to recover the context of the code cells that follow.
Right, that's the sort of thing I was talking about in terms of "using it right." It makes sense and is really a good idea, it just starts eating into the advantages you get from using Jupyter in the first place.
I'm still trying to figure out how best to integrate it in my workflow. So far I've mostly settled on using it to get some rough ideas about the data and libraries I'm using, and then just reworking from scratch in a more stable setting. As needed I'll go back and try things out there, possibly in a new notebook entirely and just do my best to copy/paste the setup over. In getting it working the setup stuff at least gets consolidated. It's... well awful but eh, I haven't really found a better way.
1) Extracting functions is harder
2) Your git commits become completely borked
3) Opening some data-heavy notebooks is neigh impossible once they have been shut down
4) Import of other modules you have in local is pretty non-trivial.
5) Refactoring is pretty hard
6) Sphinx for autodoc extraction is pretty much out of the picture
7) Non-deterministic re-runs - depending on the cell execution order you can get very different results. That's an issue when you are coming back to your code a couple of months later and try to figure what you did to get there.
There are likely work-arounds for most of these problems, but the issue is that with my standard workflow they are non-issues to start with.
In my experience, Jupyter is pretty good if you rely only on existing libraries that you are piecing together, but once you need to do more involved development work, you are screwed.
It isn't that you aren't getting good results and doing good work. It is that there are many practices that were hammered out in software development that were completely tossed out the window. 4GL comes to mind as a comparison. Makes great demos. Is not likely to lead to good engineering.
1. Interactive computation and plotting, run data through a pipeline.
2. Quick prototyping and testing. Once a function/class is ready, it goes into a module, together with some unit tests.
Nothing beats Jupyter for that second use case, 90% of the bugs are squashed through interaction and inspection in the notebook cells, not through unit tests.
Sounds like what you like is the live coding aspect. Many of the lisp environments of yesteryear would have probably appealed to you. So would many earlier math packages such as matlab and mathematica. Both of those have had environments similar to the new "notebooks" for many years.
For the pure coding side, imagine using an environment that actually allowed live redefinition of stuff in the code you had written. Step debugging. Breakpoints. Conditional breakpoints.
As a tool Excel has provided astronomical real world value.
There are only a small handful of other tools that even come close.
I think the counter is there is probably a lot of damage excel has done, as well. I'm not convinced other tools would have resulted in no bugs/problems.
This is sort of tantamount to opening a website in your browser, but as soon as it loads, you need to refresh it as to "correct" the state of the site.
That's my only compliant about jupyter notebooks. It's a powerful tool but acts a little funky if you're not used to it.
I really feel like state management with jupyter notebooks is a bit whacky. From my experience having built and maintaining websites, it's bizarre to open a notebook with a previous state but have cells not work because it needs to be reran from top to bottom.
In contrast, when opening a web page, the state is usually reflecting what you'd might expect without having to refresh the page.
I am sure there is some setting to correct this (either rerun on open or don't save state on exit), but it hurts my mental model of how a browser based application should work.
If I need a file browser or I want to quickly view two notebooks side-by-side, Jupyter Lab is still more convenient than Spyder. Although it doesn't support all the features of the notebook, and you end up installing additional extensions managed through yarn and webpack (jeez).
I never used Jupyter before getting involved in this kind of coding, data wangling and stats. And it took me a while to get used to it. The "lack of state" is confusing, as well as the apparent non-linearity of a jupyter notebook (it can appear that certain commands are done in order from top to bottom, when if you look at the number next to them, they may have been executed in a completely different order). I actually chalked this up to my background in software development, figuring I was more wired to think of state as something that would be cleared every time a script was run. Analysts (statisticians, engineers, data scientist types) like to interact back and forth very iteratively with their data, and I think they're less likely to be tripped up by some back of mind assumption that you can determine the state of variables and so forth by looking at a bunch of commands as if they were run top to bottom in a file.
It got me thinking about when I'd use jupyter notebooks for my own work. In short, if I were writing a program, even just a one page script, I wouldn't use an interactive notebook (I have opened .py files in jupyter and just used it as a text editor).
But if I wanted to "do my math homework"? A one off where I need to get an answer to some complicated questions that will require a lot of data wrangling and stats? Yeah, I'd probably reach for jupyter notebook or something like it.
Otherwise? I can't remember the O'Reilly book where this was written (sorry I wish I had the cite) but the author wrote that his favorite dev environment is python and a text editor. That's still the case for me as well.
I hate to conclude with the disappointing "the problems are when you use it for the wrong task", mainly because I've objected when people have made this argument in the past. If a tool tends to get used for the wrong task, that's partly a problem with who is using it... but we shouldn't let the tool itself off the hook too easily. It's still worth thinking through. I'm worried I'm falling into the trap of making this argument when I like the tool, and objecting to it when I don't. I'll have to think about that a bit.
As it stands, though, I haven't had a huge problem with Jupyter Notebook, and I like it a lot, probably because I don't use it when Python + vi (or a "better" but still basic text editor) would do the trick.
(The point is otherwise valid, I have seen software engineering in notebooks and it was horrific.)
As much as I love Jupyter this is a bridge too far even for me. There's nothing about Jupyter that makes sense for non-interactive scripting. Although, I suppose it would help you do block chain calculations... which would also be a terrible idea.
Interview with the author in May, 2017 - https://www.linkedin.com/pulse/why-cauldron-might-right-data...
- Notebooks are automatically versioned
- You can reference and reuse parts of other notebooks immutably through something they call "Transclusions": https://nextjournal.com/nextjournal/transclusions
- The technology-stacks underlying each notebook are immutable, meaning that notebooks work on any machine (no "hidden" dependencies).
- You can explicitly reference other results instead of relying on global state, making execution order irrelevant.
It's currently in private beta, but people are starting to use it in production. (EDIT: You can sign up using the code curryon2018)
(Disclaimer: friends of mine are building this and I used to work on it in the past.)
Am I the only one this two sentences worries deeply?
Or is it just a rehash of the twenty year old "my first exposure to programming was Excel"
(which, from what I'm reading is what the article is talking about : a glorified spreadsheet).
It's quite easy to unit test notebooks. Whenever you write a function you want to test, write some unit tests (or even just asserts) in the same cell it's defined, then any time the cell is run to define the function, the tests are run too. Likely you'll be doing some manual tests when you write a block of code cum function, and it's not hard to copy the input crafted from those manual tests to a unit test.
As many people will do this as presently write tests for their Excel macros or formulas.
-It's browser based. Yuck. This means you can wait upwards of ten seconds for a session to load up every time you want to start one. Also, generally speaking, I deeply dislike browser-based applications that have nothing to do with the internet. It just feels so clunky. My workflow gets interrupted as I have to play between windows and betwen tabs, and whenever I want to switch from a task to another I have to remind myself 'well this is my web browsing instance that also happens to be my notebook', it's not natural and gets in the way. By contrast, a terminal advertises itself as exactly what you want to use it for. It's also very snappy.
-Following up on browser-based awkwardness, my god is this thing slow. Initializing kernels. Interrupting kernels. Restarting kernels. God save you if you want to import more than a couple libraries. Wait you made a mistake and want to stop and restart that one cell? Guess you just have to go through the whole kernel thing again I guess. So you launched something and it turns out it eats way more memory than expected? Well, 'sudo pkill -9 firefox' it is, do not pass go, do not collect autosave. It just does not play nice and lags behind every thought or action you want to take. Again, contrast with a plain terminal that just does the job and does it now.
-Speaking of exiting, these things don't seem to be able to do so gracefully. If something hangs up you just have to kill everything if you want to resume working, instead of good old ctrl-C. The fact that killing everything also tends to kill the web browser you used to do actual web browsing is just icing on the cake.
-Setting things up can be dodgy. Should your notebook fail to start, error messages are extremely cryptic. Documentation for troubleshooting is lackluster. No one ever seems to have any two identical issues on Stackoverflow, which is also plagued with WorksOnMyMachine (tm) syndrome. If you ever have to mess up with both Python 2 and 3, you may be in for a good time.
-The notebook format in which sessions are saved is supposed to be human readable and editable. In practice I have never seen a single human read or edit any of these by hand, for the very simple reason they are horrible to read or edit by hand. Again, why do I have to load a file in a cryptic format in my web browser when I could just %paste my snippets off of my editor or even just use my IDE's builtin capacities (code runner on vscode, F9 in spyder, etc.)? Plus, compatibility issues between versions can make your notebooks suddenly unusable (and hard to salvage due to the aforementioned format issue). This doesn't happen with snippets pasted onto a terminal.
-IDE integration has been sloppy in my experience. I've been told this may be subject to change, however.
Note that all this criticism doesn't mean I don't respect the work of the people behind this - they've made a fantastic tool that just doesn't happens to fit my use cases, for now. Every once in a while I try to work with notebooks, find they're still unusuable, shrug and go back to my terminals. As the experience improves I may make the switch one day, who knows.
In my experience, kernels work just fine unless you are doing something you shouldn't. My kernel problems occur at points where I have a memory issues. Though I agree with you that you don't understand what's wrong once the kernel dies.
Here's the thing. I'm sure many of the complaints I voiced can be chalked up to lack of experience. But even if it weren't for the issues about documentation and troubleshooting, I'd still have yet another tool to manage, learn to use, learn best practices, etc. Instead of just getting things done with my tool it just gets in the way and tells me there are things I should and shouldn't do. It takes time to learn about it.
And now, I'm not sure the investment is worth the effort. Despite its warts, everyone uses bash and it's really easy to learn about it and translate that knowledge into many applications. And, it does the job when I ask it to. Same for Python, IDE/editor usage, LaTeX, and all the other toys in your typical academia toolbox. I'm all for learning. But time is short, deadlines loom close, and the good enough is often the better's worst enemy. And when it turns out the better isn't actually better, welp.
> It's the entire reason it's so easy to format text / math in a Jupyter notebook. There's Markdown + MathJax for basic interactive formatting, no need to compile or anything. If you need anything more sophisticated (like a box around text, etc.) just throw in a <div style="..."> and it'll automatically show up in the notebook.
> I only need one Chrome window (or more, if I choose) for my entire workspace. I can have tabs for notebooks AND webpages open at the same time, in the same program. I don't need to worry about having dozens of windows open! I also don't need to deal with some developer's idea of a tabbed workspace--Chrome's tab work exactly how I need them to.
Maybe you could get all the same benefits from an Electron app, but people don't quite like those either :).
For anything you want to have graph(e.g. matlab), notebook shall serve you well.
> While Jupyter notebooks have traditionally been a humans-only entrypoint into a program, researchers and companies alike are increasingly using notebooks for automation.
It's part of a slow journey back the power of the Lispms and the Smalltalk environments.
It's far more than scripting, and for much bigger programs than 10k LOC... My company's Python codebase is 35m LOC/500k modules, with 25k commits per week and contributions from 2.5k developers per month.
To my mind, Python has the opposite problem. I've been a Python programmer since 2000. Every few years, I think I should devote more time to other languages (first Java, then R, then Haskell, then JS, ...). But the Python ecosystem just keeps getting stronger. My current focus is data science and machine learning. In these domains, there are no good reasons to drop Python.
Can you provide a citation?
Yes some people do write scripts in Python. Does that make Python a scripting language? Maybe in your definition of scripting language.
Notable part: [..]We needed a better way to do system administration[..]
• Confusing for beginners
• Encourage bad practices
• Poor editor
Netflix said that if the job breaks they can enter the notebook with the data and see what is wrong. For me it feels like they did development with 0 safe guards and if it breaks they check why. Instead of logging problems and dealing with edge cases in the code beforehand
If you consider the notebook as a way to augment your logs with plots, it might make a bit more sense.
Running a jupyter notebook is a nice way to generate a HTML report for a job. For a typical ML pipeline, you first plot some stats about the input data, then train some model, plot some training loss, a confusion matrix, some example of predictions, etc...
If some job gives a strange result (maybe that's what they mean by break), having the notebook rendered as an HTML page with all the plot is a very effective way to do a first round of diagnostics. You can also start the notebook with the same parameters and 'run' through your report, which is a nice way to do interactive debugging.
Also in this case, the notebook itself was quite small in terms of lines of code. All the functions were implemented in modules, so it's really like the notebook is your 20 lines 'main' function. So you need some discipline among your team.