I find it so weird to see Jupyter being the "traditional" option in this discussion.
Kudos for the shout out to excel being the main "notebook" most people have experience with.
I question the choice of Python as the format. Agreed that Json is terrible. But Python will have similar problems. And a notebook is a marked up document. Using a markup language is a natural choice. Delimit areas of particular syntax and you are off to the races. (Yes, I am fond of org mode.)
Yes, fenced blocks is what I meant. Recreating parts of xml and cdata seems the common end goal. I think we have enough modern processing that you can do looser grammars.
Composition is an odd goal to me. Most notebook usage is exploratory. As such, taking effort to reuse notebooks feels counter to the benefits of doing it again that you can get.
Though, I'm a fan of seeing different workflows. And excited to be shown places I'm wrong! Kudos!
How about lessons learned from other reactive notebook environments that have been around for a while? If you are 'reinventing' something, I wonder why no mentions are made of the paths that have been laid before (Observable, Quarto, Enso, etc.).
I enjoyed the article. We are doing a more genAI-oriented notebook env variant for louie.ai aimed at database/API users preferring natural-language-first vs typical Jupyter/observable/hex/etc python wrappers, and for their automatic reactivity portion, a lot of similarity here with the road we went down. I appreciate seeing some of that reasoning laid out. Especially neat is the use of static analysis of global assignments vs the typical input/output variable syntactic convention. Few go so fine-grained (enso, frp, ...).
My personal experiences with automatic reactivity (OSS'ing the first web frp system, and many after) and last 10 years as a heavy Jupyter notebooks user with many enterprise/gov/tech DS+operational teams actually steered us down a different path. We separate out the auto-reactivity to only auto-trigger when switched into a 'live' dashboarding view. By default, when in the authoring notebook view, we stick to Jupyter-style manually triggered run / runall. When authoring, the notebook accidently running a bunch of database or OpenAI or bigquery calls when you didn't intend can be... problematic. The underlying cell-based reactivity model we went with is largely the same, but when an analyst is dealing with big database calls and returns, a world of a difference in UX.
Fwiw, if anyone uses Splunk/databricks/OpenSearch/etc and curious about a team genAI-first learning system with auto-viz-reactivity, would love to chat!
Pluto.jl is our biggest inspiration. We also took some inspiration from streamlit, and of course a lot of inspiration from the IPython kernel and Jupyter. We didn't look closely at Observable (though Fons cites Observable as his biggest inspiration), and didn't look at Quarto or Enso at all.
As others have mentioned, we're very open about our inspiration and document it in many places, such as on GitHub, though you're totally right that we could have mentioned it here too.
We could fill another blog post talking about inspiration and lessons learned from other projects — perhaps we will!
I'm a marimo author. Pluto is amazing, and definitely one of our biggest inspirations. As others have mentioned, we've given shout outs to Pluto.JL in most of our copy; oversight to not include it here.
as a daily jupyter user, i like this .py DAG approach a lot - more so for visualizing complex workflows than addressing a potentially corrupt hidden state.
there isn't much *data science functionality* integrated into jupyter core apart from displaying static dataframes (nor is their internal desire to add it), and jupyter *extensions/widgets* should really be built with dash/streamlit, so i'd encourage you to make an ecosystem.
having spent a lot of time with airflow recently, it's designed for scheduling recurring jobs, not *batch processing.* meanwhile dagster is not meant for computationally intensive/ long-running jobs and the dataset-centric approach is offputting
overengineering = since you made the cells functions already, it would be cool if you could make multiprocessing/multithreading an option in the cell UI and help keep track of multi-state variables/execution
happy to give feedback/ user interview = linkedin.com/in/laynesadler/
In principle every notebook is a DAG, since the cells are evaluated in a certain order. You could even infer the DAG from Jupyter notebooks, at least when the execution number is stored (the In[...] part).
There is a lot of value in imposing restrictions on the cells (like no reallocation of global variables) to be able to infer the DAG more directly like they do here (or Pluto.jl does). It is simpler, it can serve a dual purpose by being a normal runnable python/Julia/whatever program. Another advantage that I didn't see mentioned in TFA is parallelism. There is some interesting potential there I think.
How well does this work in practice? I am skeptical about the power of static analysis with a language like Python. Many times, the hidden state in Jupyter lives in another module, or somewhere deep in an object hierarchy.
If you’re going to reinvent the Python notebook for reproducibility, I wonder why not go further, and fully snapshot the program state?
I really want this to work, because hidden state in Jupyter notebooks bites me frequently, but the post and website haven’t convinced me this is a robust approach - maybe I’m missing something?
If you are making things public, you likely care about more than just a password, which you'll want to push into an auth gateway with oauth support, authZ, rate-limiting, etc.
Yours is the fundamentalist view and it always has its place in the debate. But it is not necessarily what the organization needs.
There are benefits of running notebooks in production rather than reimplementing things in another environment.
1. The prototype is tested and verified. And I don't mean tested as in 50 unit tests and integration tests. But tested in the way that a human actually looked at the result and was satisfied. Reimplementing means that you will start from scratch.
2. That the production env is the same as the analysis env means that it is trivial to go back and debug issues.
3. Sometimes maybe what you need is a fast solution. Not sending stuff into the scrum loop of a separate engineering team and pray it makes it into a sprint not so far away.
I am in the market for an alternative to Databricks' notebooks. Essentially, I want to use spark-connect, but also interact with delta tables directly via polars and duckdb (while still leveraging Ibis).
Support for Observable tier plots would be amazing.
similar to how you parse the cell's input variables, it would be cool if you parsed the files being read (e.g. `pd.read_parquet` and `file.open`) or databases/sources. you could show metadata about the sources next to it.
this UI setting could be turned off if every cell i/o from the same file
the only ergonomic solution to me is to maintain a converted script of notebooks and using it for diff tracking, productionizing, etc. through a simple abstraction.
Then don’t use it. This isn’t a requirement to use Python. It’s just one of the alternatives including notebooks, a text editor, or an IDE. But notebooks are very popular in some fields, and it’s nice to see new things.
no, it's a failed concept. frequently you'll find effectively shell scripts that couple workflows to some graphical environment for spurious reasons -- except they're baked with bad assumptions about state in that environment as outlined in the article.
if you want knuth's literate programming, just comment liberally per that guidance.
or carry forward with them and print your retractions later. vOv
I’m not sure what your definition of “failed” is, but notebooks of various kinds have been bringing value and accessibility to various different programming languages for a while now. If it was a “failed concept” I don’t think we’d see as much adoption, let alone new iterations and advancements in the space.
Just because you don’t use something doesn’t mean it’s not worth something to someone else (or in this case, a lot of people), despite its shortcomings.
Generally notebooks are used specifically when statefulness gets in the way, and an interactive feedback loop is the entire end goal, not the means. Being able to iterate on operations at a cell level with reactivity is totally game changing for such use cases. If you don't get where that would be valuable, you probably just haven't encountered such a use case yet.
Kudos for the shout out to excel being the main "notebook" most people have experience with.
I question the choice of Python as the format. Agreed that Json is terrible. But Python will have similar problems. And a notebook is a marked up document. Using a markup language is a natural choice. Delimit areas of particular syntax and you are off to the races. (Yes, I am fond of org mode.)