Hacker News new | past | comments | ask | show | jobs | submit login
JupyterLab 3.0 (jupyter.org)
263 points by kylebarron 59 days ago | hide | past | favorite | 140 comments

I don't write Python code for my work. Last weekend I came across an interesting Jupiter notebook and figured I'd give it a try on my work laptop. "It's probably as easy as brew install pip and then use that to load the other dependencies," I assumed.

Over an hour later I had to give up. There was initially some kind of Python version conflict on my Mac. Eventually some version of JupyterLab was installed somewhere, but it couldn't find any dependencies for the notebook. As a complete newbie to the Python ecosystem, I googled for instructions and found various environment managers, which then failed possibly because something is incompatible with Big Sur. More googling revealed instructions with complex CFLAGS environment setups to fix it, which didn't — and at that point it was very far from the supposed convenience of scripting languages anyway.

I don't think it's Python's fault. Probably all the programming toolchains are this hard to a newbie! But it was a humbling experience after 34 years of programming, not being able to load a piece of sample code in a Sunday afternoon.

> I don't think it's Python's fault.

Well, there's a long legacy of half-baked solutions and kludges. And, especially, install instructions consistently suggest you do the worst thing: install stuff using pip.

Everyone knows that's bad advice, because it always fails in mysterious ways, and there's no good way to roll everything back.

I recommend:

    python3 -m pip install pipx  # Don't rely on the pip3 bin.
    pipx install jupyterlab
    # Now jupyter has an independent virtualenv.
    # To install dependencies:
    pipx inject jupyterlab numpy
    # Where is everything?
    pipx list
    venvs are in /Users/ben/.local/pipx/venvs
    apps are exposed on your $PATH at /Users/ben/.local/bin
Once you've done that, never install anything directly with pip3 again. If you're working on a project, use pipx to install poetry / dephell / pipenv and use them to manage the virtualenv.

And you can get rid of it all:

    pipx uninstall-all
    python3 -m pip uninstall pipx

I use conda to do effectively the same thing. If my conda environment breaks and I can't roll it back, at least I can destroy it and start again.

As an aside; it is really hard to explain why you should use anything but pip3 to people who only rarely need to interact with python.

Conda is pretty nice, and unless someone needs pypi, that's fair advice.

And I agree, it's hard to explain not using pip3, so I just wouldn't. "For the best experience, pipx install neato-commando." And maybe link to the pipx website, in case someone actually clicks a link.

Conda works fine with Pip/PyPI.

Interesting coincidence: I'm currently side-watching a Youtube video about poetry.

The video discusses python packaging in general. You have to skip through several minutes of pre-roll though. https://www.youtube.com/watch?v=tNlurLxcf68&feature=emb_logo

Yup this is a good way to go around it, similar to another comment below, I use conda to manage all the stuff

As someone who has done driveby python work, how would I even know how to do that? Noob me found all sorts of howtos and getting started guides. Seemingly contradictory, mutual exclusive.

My (extended) team couldn't even get all the Pythonistas to agree on The One True Python Stack™.

Following Python and Java is always informative because, between them, they manage to make almost every mistake imaginable, despite having very smart people putting careful thought into the design.

Looking at the amount of polish on the Python docs and the fact that people still don't know how to install Python, it gives you a sense of how hard explaining things is. Maybe the Python site could do with an official "managing Python for end-users."

> Noob me found all sorts of howtos and getting started guides. Seemingly contradictory, mutual exclusive.

Python's a victim of its own success as the signal to noise ratio in online tutorials is very low; worse, the good management tools are relatively new.

Even for smaller audiences, obsolescence is a big problem. The most painful aspect of working with LaTeX was I'd struggle with one buggy package recommended by a tutorial for ages, then search for "why am I getting this error" and find the answer was, "oh, yeah, that's horribly broken, use this one instead."

[1]: https://www.python.org/doc/sunset-python-2/ [2]: https://www.python.org/doc/versions/

You don't. It's a total nightmare for newbies.

I ended up putting off learning Python for years as this kind of stuff kept happening (first the 2/3 transition, and then massive pip breakage).

Thank you! I've been using virtualenv+pip, conda+pip and a few other virtualenv wrappers lately, and never heard of pipx!

I have used Python on OSX for years and it is and always will be a horrorshow. Using the system Python installation is a nonstarter for many reasons, chief among them is that I don't have any interest in using py2. So then you're using pyenv or homebrew, but your vim install still thinks that it should be using the system python. And whoops, you fixed that and now virtualenv is not finding your interpreter. And etc., etc.

OSX and its tooling are just ridiculous. I have no idea to this day how macs became the premier development environment.

As always xkcd has a comic for it: https://xkcd.com/1987/

> I have no idea to this day how macs became the premier development environment.

I've been editing a tutorial one of my coworkers wrote that targets new Python users on Windows. From my findings, the grass is not greener.

Granted, geospatial Python is somewhat of a mess, but a lot of tools I have to use are somewhat messy forks of Unix tools (looking at you, pyenv-win) with tons of incompatible extensions. For development, Windows is the exception because you can transfer just about anything from Linux to macOS.

On my Mac, I can easily install all of the Python packages I need without needing to install Visual Studio, pipwin, anaconda, etc. I have bash/zsh as my default system shell. Maybe it's easier to native Windows users, but bash/brew is a much better combination than anything I've found in Windows.

WSL is a step in the right direction, but it still feels secondary. If a first-class terminal experience existed on Windows, I have a strong feeling that it could be the premier development environment, or at least closer to Mac.

Maybe I'm using it wrong, but it definitely hasn't been made clear on how to use it right.

> I've been editing a tutorial one of my coworkers wrote that targets new Python users on Windows. From my findings, the grass is not greener.

Maybe check the other other side (Linux) - I found the grass is greener there - at least for Python (and programming tools in general). I'm very comfortable on the command-line, and moving from a pure Linux environment to OS X & brew felt like a huge downgrade, followed by random annoyances that remind you you are using inferior, non-GNU utilities:

  $ ls my_dir -l
  ls: -l: No such directory
Really - Mac OS? I know its minor, but that's just user-hostile and it happens every few weeks; I can't get over it.

>followed by random annoyances that remind you you are using inferior, non-GNU utilities:

Well, you can switch to another ls in 10 seconds by "brew install gnutools" or some such.

Not to mention the same arguments could be made for FreeBSD, commercial unices, etc.

Come to think of it, I've been using Unix (including Linux) for 25 years, and never even occured to me to expect "ls my_dir -l" to work.

What do you do if you want to add extra arguments when your cursor's at the end? Navigate back near (not to) the beginning? I usually just put them at the end, as almost every program I've used supports it. I add extra arguments after running commands constantly; after pressing up to get it back, you're at the end.

>What do you do if you want to add extra arguments when your cursor's at the end? Navigate back near (not to) the beginning?

Yes. In most shells it's trivial to go back by a word (e.g. skip in front of the path with one shortcut).

That said, this happens so rare that it's not even something that ever registered with me as a problem. I know what arguments I want for ls. And if I don't it's usually some bizarro infrequent flag that I have to look up how it's used anyway.

And other programs where it would be actually useful don't allow it anyway (having flags after the "main" argument). Git, for one.

> Well, you can switch to another ls in 10 seconds by "brew install gnutools" or some such.

I did - but I never remember to type "gls" instead of "ls" - if I were that mindful, I'd always put in the flags at the start, where the BSD utils expect them. As sibling comment noted, trailing flags usually happens when I run a command and decide to amend the flags by up-arrowing and typing. A typical result is "ls -al my_dir -tr" when I realize I want to sort by mod date, post-facto.

I've been using GNU utils 95% of the time I've used Unix(-like) systems: I expect "ls -al my_dir -tr" to work: it's 2021 - recognizing flags isn't rocket science.

>>>> I've been editing a tutorial one of my coworkers wrote that targets new Python users on Windows. From my findings, the grass is not greener.

I've been using WinPython for a few years. It's the closest thing to "just works" that I've found, and non-programming colleagues have been able to install it successfully.

Because it works almost like an isolated "container," it's also possible to test your code on a clean install of WinPython to make sure it will run on someone else's computer before you share it.

I don't know the technical difference between WinPython and a true container, but you can have multiple WinPython installs on one machine, and they don't interfere with one another, or with a pre-existing mainstream Python installation on the same PC. So you can share your stuff without worrying about screwing up someone else's stuff.

Good to know, thank you!

>OSX and its tooling are just ridiculous. I have no idea to this day how macs became the premier development environment.


(a) it's quite easy to set things up with brew, macports, and/or Nix

(b) because Python is shitty everywhere anyway, and Python isn't the be-all end-all of development work.

(c) because you get a full-featured, working, coherent, take-it-or-leave-it desktop that stops one way of endless tinkering and procrastinating available in Linux to get things "just right"

(d) because it's still a UNIX with a full support for unicy tools, not a hack like WSL or WSL2.

(e) because it has good hardware (mostly - BS keyboard-era aside) and good resale value

(f) because you get to enjoy most/all the proprietary tools you like too (from MS Office and Adobe Creative Suite, to whatever)

(g) because in 2020 Docker, remote environments, etc, make many "local dev environment" points moot anyway

I think it's more inertia than any of these things.

The entire point of this thread is that (a) is false -- see grandparent and the xkcd joke. It's not easy. It pretends to be easy, but is usually broken in some crazy way instead. Apt is also easy, but it actually works more often than not.

(c) was relevant in 2006, when the novelty of OS X was that it was a UNIX that you could actually use as a daily driver. This is what initially got developers to move to Mac. But it's been fifteen years, and all jokes aside, the "year of the Linux desktop" for developers was probably around 2012. Linux may still have issues, but they're not worse than the hoops you have to jump through to make today's macOS behave.

Trendy tech companies are still buying macbook pros for their employees because that's what's been trendy for the last decade, not because they actually ask new hires what they prefer. Practical tech companies do, and at places like that you usually see a mix of macs and thinkpads.

>(c) was relevant in 2006, when the novelty of OS X was that it was a UNIX that you could actually use as a daily driver. This is what initially got developers to move to Mac. But it's been fifteen years, and all jokes aside, the "year of the Linux desktop" for developers was probably around 2012. Linux may still have issues, but they're not worse than the hoops you have to jump through to make today's macOS behave.

I've used Linux for close to 20+ years, and Unices more, and never had to jump through any major hoops to make macOS behave.

What would those be (talking about something major, not "I can't get my favorite window manager to replace the macOS window management" -- the non-tinkering-friendliness is part of the allure to me and from what I read others too)?

On the other hand, Linux on the desktop never fails to dissapoint me in one way or another because of the need of tinkering, half-sketched apps for many things I want to do (especially anything multimedia and/or document related), driver issues to get things working (sound, compositor, 3D, bluetooth, sleep, etc), and so on. And judging from the everpresent "just use <name of another distro>" in the relevent forums, it's not something others don't have.

Thus I prefer to stick to Linux on the server and Docker, or for setups where I have investigated the hardware in advance, and only mean to use basic things (e.g. happy with just some terminals, emacs/vim, i3, and some mp3 playing).

>not because they actually ask new hires what they prefer.

Those that do found that hires generally prefer Macs. That's how they have ~ 50% of the dev surveys on Stack Overflow whereas they're just 10% of the general market...

>was relevant in 2006, when the novelty of OS X was that it was a UNIX that you could actually use as a daily driver. This is what initially got developers to move to Mac. But it's been fifteen years

My anecdata for this is that I bought a MacBook in 2020 for precisely this reason. I am not a programmer, mostly my use case is bioinformatics and processing large datasets. Native terminal is better than WSL (although WSL is good now) and macOS is much more reliable than linux ever has been for me.

b and g are mutually exclusive, though?

Not in any logical sense (e.g. violating any Logic rule). They just cover different use cases.

(b) makes the "macOS is particularly bad for development because I found Jupyter/Python deps difficult there" argument moot, as messed up Python dependencies are the case in Windows and Linux as well.

And (g) says that Docker and co has superceded manually setting up Python environments for many (not necessarily all or even most) devs, mitigating concerns about managing multiple local versions different deps/libs/language versions to work with different projects (since you can now do that in different, isolated, virtual environments which are mini-OSes in themselves).

So (b) basically amounts to: "It's not macOS which is makes Python deps shitty, they are inherently shitty".

And (g) basically amounts to: "Since Docker and co make local development dep issues mostly obsolete, even if macOS was bad at local deps, it wouldn't matter as much today anyway as virtualization levels the field".

And of course, with the field levelled by (g), if you go for virtualized dev envrironments, you still get all the other benefits like e, f, c, and d.

> I have no idea to this day how macs became the premier development environment.

    s/development environment/scripting language

True. I like Python, but the ecosystem can be very confusing for a quick dive. It's easy to get lost between pyenv, virtualenv, pipenv, pip, pip3, easy_install and friends.

Very true - further evidenced by the five or six different approaches your sibling comments are recommending.

Don't even try. Use docker, there are official pre-built images. It's easy:

  docker run -p 8888:8888 -v ~/Code/actual-project/:/home/jovyan jupyter/datascience-notebook
Note ~/Code/actual-project/ is a git repo or a mounted Google Drive folder. Moving from one machine to an other is pain-free.

[1] https://jupyter-docker-stacks.readthedocs.io/en/latest/using...

PSA: Don't use docker run -p xxx:xxx lightly. Even (or especially!) not in examples and documentation. It will listen to connections from anywhere in the internet and go out of its way to manipulate firewall rules on your machine to make a hole for it.

Instead, use: docker run -p

At the risk of contributing my own probably-subtly-broken advice to the collection already present in this thread, I would note that there is a relatively sane system python3 on Catalina/Big Sur, and most projects are building and distributing compatible wheels [1] (binary packages). So something like:

    /usr/bin/python3 -m venv ./venv # create a virtual environment in ./venv
    ./venv/bin/pip install wheel # optional
    ./venv/bin/pip install dep1 dep2 dep3 ...
    ./venv/bin/python ... # up and running
is enough to get started in most cases. Of course, all bets are off for GPU or Apple Silicon support...

[1]: https://pythonwheels.com/

In a better world, the notebook you downloaded should have come with a list of dependencies and have reproducible code in it. That said, Python environment tools and package managers are still in a state of flux. Some tools make it easier, but are not helping much with reproducibility (plain venv + pip, but only if you got all required system packages installed!), other tools enable reproducibility, but have growing pains and are more difficult to use or get working (pipenv, poetry).

If you are in luck, your required libraries and appropriate versions are available as GNU Guix packages and you can use that.

I talked to Fernando about this at Neuroinformatics 2018. In principle generating dependencies from a notebook should be as easy as parsing the ast, finding all the import statements, and then using sys to lookup what package provided that module. In practice I stopped using notebooks entirely due to many of the issues described by pavlov above, though in my case it was mostly because it was at a point in time where the Gentoo python ecosystem was having a rough time dealing with the 2/3 transition and there was no minimal amount of work that could be done to get the jupyter server up and running.

As for the reason why this still hasn't happened? I'm guessing because it is because the people who care about reproducibility have found other ways to achieve it, and as a result have no need nor incentive to implement it as part of the notebook format. Also because, if someone is running notebooks at all they know enough to bootstrap the jupyter server, which is non-trivial, or they are using a managed environment (be it remote, conda, etc.) where they are blissfully unaware that all their code has massive implicit and undocumented context.

In the mean time I have implemented the start of an equivalent system for org files because the implicit environment is much worse given normal Emacs workflows. I have encountered some significant issues and considerations which the notebook ecosystem would have to deal with, and which the default python packaging ecosystem (or any ghettoized language specific packaging solution for that matter) is not equipped to handle.

The primary issues is that only language agnostic package managers can actually bootstrap the environment that is required. What do you do if someone doesn't have a toolchain to build numpy and they are on some strange arch where there are no wheels? Do you just fail? The 80% solution here is the one we have with conda or managed solutions. The 90% solution requires that all the language communities stop pretending that they are the only language in the world and stop wasting time on their bespoke package manager.

> In principle generating dependencies from a notebook should be as easy as parsing the ast, finding all the import statements, and then using sys to lookup what package provided that module.

Well not really.

First, there is no link between a distribution package name (what you install with pip) and python package name (what you import).

e.g. The distribution package "foobar" can install the python package "baz". Well, it's even worst than that. There could be name space packages so multiple distribution package actually install parts of a python package. And there could be monkey patching.

Second, Python is not a static language. The import system is part of the interpretation of python code, and as such, is dynamic.

You have absolutely 0 guarantee that "import foo" on two different environments, even if they have the same version of package foo, will produce the same results. There could be an infinite amount of magic done in foo's __init__ that depend on other libraries being installed, etc.

Third, don't get me started on the actual version of these libraries.

I agree on some of your points. I just want to add:

Then comes the point, where one of your dependencies in conda simply wont install on the machine of your coworker or a windows machine of another coworker, or in your docker image and you are screwed, scrambling for the next package manager. Been there, experienced that.

I can only recommend to people not to go that route and to not get into trying to get the same environment on windows. A lot of headaches. It will be simpler to tell people to use another OS, if they want to use the same environment. Ultimately I ended up providing a virtual machine.

A managed solution -- Yes, this takes a lot of load off of the shoulders of people, who would like to actually get productive in other ways.

> Python environment tools and package managers are still in a state of flux

They've pretty much always been a moving target, haven't they? Nothing has ever been good enough that the community agrees "okay, this is it, we can stop churning on this tooling for a while".

Probably because the typical culprits are really not good enough. I mean, follow the package manager repos on Github (pipenv for example) for a while. They have loads of issues, which one would think are unthinkable for a package manager to have. There are regressions and sometimes environments that were created just fine in an older version, is not creatable any longer, even though all the specification is, is a bunch of package names and version numbers, which do actually exist.

Probably testing package managers is not a simple thing to do.

can look into google colab to directly load the notebook. Link : https://colab.research.google.com/

Python is AMAZING and Awesome TILL You want to spread what you are doing to more then one computer. Though I have to say I love domain specific languages and personal enjoy Racket as my most fun language to use. It is sad that this has gotten so much worse over the years.


  A: it works on my computer
  B: then we'll ship your computer
  and that's how docker was born

What do you enjoy about racket?

I used it for a semester in college, and it was next to unbearable. I can appreciate it as a fun little functional programming gimmick, but nothing more than that.

I just love Lisp and you pretty much have a fun small language you can fit in my small brain with Racket. The biggest issue is the libraries. Everyone writes their own specific libraries for their problems, because its easy and fun.

I could have done it in Python but portability was always the issue.

At my former job I had to make charts from pictures with English and Spanish lettering. Took me 30 minutes to have a working client that got installed in a dozen computers that afternoon.

I needed to also re-structure the companies file system for every computer in our company.

Personally I worked in audio video contract work from time to time and I was able to make a dozen short cut programs that actually saved my biscuit several times. I could have done it in python but the portability kicked my butt. Racket made executables super easy.

Also I went through How to Design Programs and I learned a ton.

> I don't think it's Python's fault. Probably all the programming toolchains are this hard to a newbie!

Python is a little worse in that in pushes its dirtiness downstream to your users too.

Not sure about the Jupyter-specific issues but macOS comes with Python 2 installed, so you have to install Python separately and use the `python3` binary, or alias `python` to the `python3` binary.

Then remember to run pip using `python3 -m pip <command>` and use virtual environments (venv)

I stumbled on this a couple weeks ago and documented the process here https://flaviocopes.com/python-installation-macos/

I found the same problem with Ruby (old version preinstalled on macOS)

You should use Anaconda instead of pip to install data science related tools in Python.


$ conda create -n {new_environment_name} python

$ conda activate {new environment name}

$ conda install jupyterlab


You also have to manually add the kernel for that environment via:

({new_environment_name}) ...$ python -m ipykernel install --user --name {new_environment_name} --display-name "Python ({new environment display name})"

The env won't be accessible in jupyter (lab or notebook) until you do this step.

[0] https://stackoverflow.com/questions/39604271/conda-environme...

Nope, what he wrote works just fine at least for conda. That advice is only if you want to be able to choose environments as kernels in the frontend. If you just run jupyter lab/notebook in a conda env which has it installed, the default kernel will be in the activated environment.

Note that you now have to pay for a commercial anaconda license if you're using it for anything that isn't education or hobbyist stuff.


I don't think that is true if you install miniconda.


    conda install -c conda-forge jupyterlab=3


I'm in the same boat. That's why I use repl.it[0] for nearly all my programming now. Haven't tried jupyter on repl.it, but here's[1] the first DDG result, which suggests jupyter is supported.

[0] https://repl.it/ [1] https://repl.it/talk/templates/Jupyter-Notebook/49993

Funny. I wanted to try this so I just ran the docker image and it worked. Time spent: 2 mins.

docker run -p 8888:8888 jupyter/scipy-notebook

Had same experience.

Hoping to help clear some backlog, I tried to help with some ML & data pipeline house cleaning. Had never used Python before, but how hard could it be? OMG. One really doesn't appreciate ease and simplicity of multiple simultaneous JDK installs, all peacefully coexisting, until working on some python and nodejs projects.

Just running user space dev env via Docker was recently front paged here on HN. I believe that will become the norm.

Being a macOS user, I looked to see if there was something more lightweight. Like maybe a port of podman. Lighter than running a virtual Linux instance.

Again as a macOS user, I feel like I'm overdue to embrace the remote dev desktop set up. I've read that Googlers don't do any work locally; they all remote shell into servers running the tool stack. So very X Windows; what is old is new again.

Always try using a virtual env. Also try using pip directly through ‘python3 -m pip install jupyter’ . Same to run it to bypass path issues ‘python3 -m jupyter notebook’. You should be up and running in a couple mins!

These were among the suggestions that I tried. I think the virtual env failed because of Big Sur, but not sure.

Hmm that’s super weird, M1 or Intel CPU? Also where does ‘which python3’ point to?

I find that everything in Jupyter-land is as inconvenient to reproduce as it is convenient to experiment with. Case in point: last I checked, you still had to do gymnastics in order to even store your notebooks in Git in a meaningful way (e.g. allowing merges).

It really throws me off the whole thing (Julia + Jupyter is even worse...) because I’m not keen on putting effort into stuff that nobody, myself included, will be able to use in a year or two.

For Julia you might as well use Pluto (https://github.com/fonsp/Pluto.jl) or an IDE.

Thanks, I’ll check those out! It’s hard to know what’s recommended by the community sometimes. I was using Julia + Jupyter through CoCalc which is great for collaborating but the non reproducibility aspect was starting to raise alarm bells for me.

My most recent professional project involves - among other things - a Python API deployed to AKS. I'd used Python here and there for fun and on the periphery of work, but this was the first time I had to dive into it full-on.

While I know it has earned its praise for good reason -- as I'm sure all the data scientists I work with would attest to -- as an engineer, Python has been really frustrating to work with.

Yeah, I use portainer and then load a docker image of Jupiter and import the notebook there.

Dependency management hell isn't just a Javascript problem these days.

Yeah I've had this issue in the past with Jupyter. Now I generally look for docker images when I want to check out something new.

>I don't think it's Python's fault.

Actually it is:


That said, you would be up and running if you used conda.

And this is not the usual "just try distro X" Linux advocate BS when somebody has an issue with distro Y.

Conda does makes thing more user friendly by focusing on whole environments. Actually that's why it was built for.

Note that this applies for this situation. For other uses cases, e.g. deploying something to production, having used conda would have made your life more difficult.

When following the docs, Go just works. It compiles fast, though not so dynamic as Ruby and Python. Tooling and runtime also matters in one's work.

Not sure about Mac, but on Linux, you install some docker images and it just works (modulo some docker flags to learn if you don't already know docker)

Relevant XKCD.


No problems on Linux. I use pyenv for managing python versions and virtualenvs.

That doesn't help you install the dependencies for a notebook, if you don't know what they are.

virtualenv makes things a breeze

Ok this may be kind of stupid, but my primary reason for not switching over to jupyter lab (from notebooks) was the right-click menu wouldn't let me copy images (plotting outputs typically) from the in-line output. However I found out today that it has always been available if you hold shift...so if that was anyone's issue this is a great time to give JupyterLab another shot!

Ugh, that's a lot simpler than my solution. I found out (I think through SO) that if you select the 'Create New View for Output' option, you can copy and paste as normal.

That was an explicit decision in the underlying UI toolkit, which supports overriding the context menu in a generic way, to move the browser's default context menu to Shift-Click.

I'm one of those extension developers that working on getting my extension working on JupyterLab 3.0 :-)

I've been developing Mito (https://trymito.io), a spreadsheet extension to JupyterLab that allows you to edit a spreadsheet in a notebook as if you're editing Excel. You edit the spreadsheet, and Python code gets generated that corresponds to your edits.

Feedback on the above greatly appreciated. And congrats on the release :)

I'll have to try that next time I'm struggling with pandas.

One thing to look into is figuring out some logic of cancellation to clean up the generated code. Even if you just checked back one step it would mean less Foo_Bar['D'] = 0.

I think saving the analyses in the home directory is defeating the purpose. One appeal of a notebook is that you can break all the steps out and show them to someone, add commentary, etc.

I'd want to use it interactively to figure out how to get my data looking the way I want, and then show off how the analysis works by pulling out non-interactive Step widgets that visualize changes into their own cells.

That said, I'm often doing stuff with numpy so I need the notebook to document exactly what the dimensions mean, why I did a "clever" thing, etc.

Also, as long as "import mitosheet" is hitting your analytics APIs, it's going to set off alarm bells if we try to use it inside our VPN. The compliance people are not likely to look at what it's sending or why, they'll just ban it and move on.

Looks nice! Which spreadsheet/grid library have you used, as I've not found one performant enough once you get 1000+ rows?

Just to say a big thank you to all those who have worked on JupyterLab. It's by far my favourite notebook interface and I think gets the balance between a clean interface and powerful features just about right.

I just wish that the cloud providers would adopt it as the basis for their products.

It’s not a basis for their products but this is something https://colab.research.google.com/notebooks/intro.ipynb#rece...

Jup, MS Colab is one of the many commercial Jupyter/IPython notebooks available. There are also Y-combinator startups like https://deepnote.com/

I hope that these closed source commercial platforms give back something to the open source community around Jupyter once they get out of the red numbers.

Indeed and they all use their own notebook interface which are (in my view) inferior or at least no better than JupyterLab.

I get that they feel the need to add their own features (collaboration etc) to distinguish themselves but it would be so much better if they had settled on a single UI.

We do use JupyterLab for the notebook experience on https://iko.ai. I had a chat with one of the core members of the project using real-time-collaboration when they were trying iko.ai. We chatted on the notebook itself.

What I explained was that in our experience of many years doing paid machine learning projects for large enterprise, we never thought to ourselves: "Darn, if only Jupyter[Lab] had better stylesheets, this project would go so much faster." It has never happened. When we started building our platform, we were not in the position of front-end devs trying to make a better notebook, we were in the position of a company with deliverables for yesterday, and we naturally started solving for actual problems doing machine learning for real clients, as opposed to finding windmills to fight.

In our experience, projects are slow not for lack of better stylesheets or animations. Therefore, we ignored that and focused on removing frustrations we had in the real world: long-running notebook scheduling, automatic experiment tracking for parameters, metrics, models, and code. Model deployment and monitoring. Real-time editing.

That's why I don't really follow news about "jupyter-killers". I kept an eye out of curiosity, but every time one pops out, they solved things that don't matter much, or claimed they solved hidden state until you read the article and found they're caching results or something like that.

Thanks - I've just signed up for an account and it looks really good! I'll say hello on Slack in a moment.

We've been working on a JupyterLab based product for the insurance industry - not machine learning but shares a lot of the features - e.g. possibly long running models etc. Trying to land our first (fairly large) customer at the moment!

Can you share any more on what your plans are especially from a business perspective?

>We've been working on a JupyterLab based product for the insurance industry - not machine learning but shares a lot of the features - e.g. possibly long running models etc

Cool. Do you have a link I could check out? I exchanged with someone a few days ago who is working /has worked in the insurance sector. Could you send me your contact information or add it to your profile? I'll ask them if they're interested in getting in touch with you or take a look at your product.

>Trying to land our first (fairly large) customer at the moment!

Nice. We're coming at it from the other way: we've had organizations as clients for our consultancy and we built complete custom machine learning solutions for them [as in, sometimes doing 3D printing to serve data acquisition, writing libraries for BLE devices, to the models, and web/mobile apps, as they wanted us to completely handle everything]. After doing that for many years, we clustered problems we consistently ran into, and then decided to build our internal platform to help us deliver better.

>Can you share any more on what your plans are especially from a business perspective?

First, keep our sanity. It feels really good to see all the tedious work we don't have to do anymore on the consulting side because of what our tool does for us. We continue to do consulting to keep our finger on the pulse, plus the problems we solve are really cool and diverse.

We're very, very niche so we don't even have a presence on the web yet - route to market will be through corporate advisors so it's not something we'd expect clients to sign up for online etc. Very happy to do demo or share more information though. I'll send contact details on Slack.

The application is to do long term financial projections for life insurers. Currently market dominated by applications such as FIS Prophet and Risk Agility - but we believe that we're a) faster and b) better integrated with other data / analysis tools. These jobs can be enormously compute intensive (eg see [1]) so being able to do them faster / with less resources can be very valuable.

Glad you're keeping your sanity. Very impressed with the utility of what you're doing.

[1] https://integrate.milliman.com/-/media/integrate/pdfs/azure-...

I was really excited to see the visual debugger there. I tried it myself, it was really easy to get it going. Unfortunately, using it briefly in the notebook I was using I found it pretty buggy, lots of glitches and big slowdowns in the UI. I would also really love to see the option to drop into an interpreter while debugging.

Still, can't complain too much about an open source project, thanks to the team for all their hard work.

I don't understand why Jupyter notebooks are still in use as a writable format when there are editors like VSCode that can treat ordinary python files as notebooks. The Python extension can submit code blocks to the kernel and import and export Jupyter notebooks, but VSCode is also a real editor on top of that (debugging, git, vim mode, hover info, etc.). For distribution, as a read-only format (like PDF), notebooks are great. But why do people continue to do their work in one?

I don't think anyone who writes production code is using Jupyter as a development env. It's more like a self-documenting, persistent repl, or perhaps an interactive doc.

I've used it as a way to document data exploration for later use and as an interactive teaching tool. For that, being able to plot inline and embed interactive widgets is great. Also, mixing in rendered markdown content.

If you're using Jupyter to write python software and the software is the end result and not the data analysis/process, I think you might be doing it wrong.

Not that I'm discounting data scientists (or the like) who use it as their primary environment. Their end result is not a packaged/deployed software and the code itself is just ancillary. For those users, the code and environment is a tool, and the extra weight of what a full IDE (like VSCode or PyCharm) and the features they bring to the table are unnecessary and likely counter-productive.

> Not that I'm discounting data scientists (or the like) who use it as their primary environment.

Unfortunately it often ends up needing to be executed as a batch job or integrated into a pipeline, which then starts the fun job of extracting the code from the notebook piece by piece.

Jupyter having a plain text file format instead of json would make life so much easier, and vastly improve version control.

For batch jobs you have papermill, for plaintext format you have the jupytext extension that pairs the Jain notebook with a script with notebook markup in comments. I honestly wish that they had used that format as just being the notebook format, but it’s a nice enough and easy fix.

Not Netflix, but we do use notebooks on our internal platform. Long-running notebooks for model training, but you can view their state as they run even after you close the tab or get disconnected from the internet.

We also publish AppBooks, which are automatically parametrized notebooks. However, we automatically detect and track experiments, and then we deploy models in the form of an endpoint to be invoked by consumers, with a convenience page for humans to invoke the code by sending JSON data, or uploading a CSV file.

- https://iko.ai/docs/notebook/

- https://iko.ai/docs/appbook/

Maybe I misunderstand something, but I use jupyter for sage for example and the point of it is that graphical results can be shown there immediately. Some plots can be even made interactive.

When running a Python kernel, VSCode splits the editor into two panes, the input .py file pane and the output ipython pane. The output pane supports interactive graphs just like Jupyter does.

1. That's still less intuitive for someone who wants to understand the relationship between the code and output.

2. Following from 1, notebooks are meant to be shared with output in place, so others can understand without executing code.

Exactly. When you want to show results to a client, for example.

>2. Following from 1, notebooks are meant to be shared with output in place, so others can understand without executing code.

We wanted to let others execute the code without being overwhelmed by the code, but we also wanted our colleagues not to write an application or parametrize the notebook by adding metadata or tagging cells, so we added a feature named AppBooks[0]. We automatically detect parameters you want to expose, build a form on top of your notebook, and then serve a clean page.

The user gets a page with a form with fields corresponding to your parameters, they just change the values and hit Run. The notebook runs with the new parameters and the client/stakeholder gets the result.

This solves the problem of: "I have a notebook, how do I not only show my work, but allow domain experts to tinker and run this with specific values". Especially useful when the parameters are pretty domain specific (like instrumentation for nuclear power plants where the client wants to try the model on edge values).

These runs are tracked in case you want to share a training AppBook where it produces a better model, for example.

- https://iko.ai/docs/appbook

> why do people continue to do their work in one? Most nb users don't use vscode. I'd say main reason is you can't open a notebook without starting a kernel which takes >1 second. It also is harder to setup.

In terms of a flat file formats, there is actually a mark down extension which I would say is better than vscode's format because it is more standardised:


just discovered 'jupytext', which is also pretty good for this: https://jupytext.readthedocs.io/en/latest/

> when there are editors like VSCode that can treat ordinary python files as notebooks

was doing this in emacs before there was a jupyter notebook..

as the name implies, notebooks aren't for development, they are for interactive, shareable presentation, and presentation that also works over HTTP without needing local tooling

I think you're suggesting that if you want to retain both code and results you need to manage two different files - raw python and exported notebook. I think many notebook users would see that as inconvenient (and many are not familiar with VSCode).

Jupyterlab has an extention that does this automatically on save, jupytext, also keeps both files in sync over time.

I use PyCharm + Jupyter Notebooks with autoreload. I'll build up functions in the notebook by running small bits of code, then move them to a Python file. It's like a supercharged version of a REPL.

VSCode's version is nice for sure, but not nearly as nice as being able to go back and read/modify old cells.

I skimmed the page and didn't see any mention of sharing or collaborative use. That's the biggest obstacle I'm seeing with getting buy-in at work. I need to be able to let some users see the notebook in read-only mode, others should be able to run it but not edit it, others should have full access.

Maybe there's a non-hacky way to do this and I'm missing it?

Our solution to this is just to publish notebooks to an internal gitlab repo. Easy to share read-only versions, it renders a bit different but looks close enough. You can then easily manage user access rights in gitlab and contributing back changes is easy with the jupyterlab git integration.

For publishing interactive examples we use voila, which creates a dashboard version of the notebook for users.

jupyterhub addresses some of these concerns.

I've seen it crop up here and there. I've used it a few times and its really amazing how fast.

Afaik it's a drop in replacement for conda using the same cli parse. Even has miniconda (micromamba).

Did you reply to the correct comment? I can't understand what you're referencing here.

I think they're talking about mamba and replied to the wrong thing.

Interesting that they include mamba install instructions before conda.

Is mamba taking off with the recent (and upcoming) license changes that anaconda made? I’m not familiar with mamba though I’ve had my eyes open for an alternative to anaconda (also seen poetry mentioned a few times here).

Great news! I'd only just updated the Purple Please theme[1] for Jupyter Lab 2.0, will have to look at doing the same for Jupyter Lab 3.0. It looks like they've made some improvements to extension development too which is good to see.

[1] https://datacrayon.com/posts/tools/jupyter/theme-purple-plea...

Hmm, what is mamba? I thought I was at least sort of up to date with the various python dependency managers, but that one is new to me.

Relatedly, I have a grand vision of having as part of my development environment a jupyter notebook always at hand, in which to explore data as necessary, whip up scripts, accumulate little helper functions, etc. Basically, any time I have that "hm, I wonder..." itch, I'd like to be able to quickly whip up a solution in my notebook. I'm a skilled developer in terms of larger systems, but have a weakness when it comes to very early stage "throwaway" scripts to answer ad-hoc questions.

I spent a few days trying to set up jupyterlab in an "ideal" way, so that I could have easy access to python libraries within my notebook, and also a reproducible environment since I plan to push my working directory to GitHub and would like to sync across different machines. I got confused by virtualenv vs venv, and tried conda to install libraries, but ran into various problems. Oh, and part of it was trying to have nbdev (from the fastai folks) as part of this toolkit.

Anyone have a setup like this that they use and want to share? Part of me wants to just throw in the towel and learn R and RStudio, since I've heard its ggplot is the best plotting library anyway.

I use a setup like this. Essentially, I use pyenv [1] to manage Python versions and Poetry [2] [2](https://python-poetry.org/) for virtualenvs/dependencies.

The workflow for creating a new project looks like this:

1. Create a project directory (e.g. 'myproject') and `cd` into it. 2. `git init` 3. Fixate the Python version for that project with the `pyenv local` command (e.g. `pyenv local 3.8.6`). This creates a `.python-version` file that you can put under source control. Within the `myproject` directory tree, `python` will now be automatically resolved to the specified version. Your system Python (in fact, any other Python versions you might have installed) remain untouched. 4. Create a new poetry project (`poetry init`). This creates a `pyproject.toml` which contains project metadata + dependencies and can also be checked into git. 5. Add dependencies with `poetry add`. Here, you could for instance add Jupyter Lab (`poetry add jupyterlab`).

To access installed dependencies, such as the `jupyter lab` command, you can either execute one command in the virtualenv directly (`poetry run jupyter lab`) or spawn a shell (`poetry shell`). If you open a Jupyter Notebook that way, the packages installed in the virtualenv are directly available from within Jupyter Notebooks, without having to mess around with installing IPython kernels.

I like this approach, because it gives you full flexibility, while being portable and easy to use. It gets you around having to deal with conda (which I found to be frustrating at times). Also, you're not tied to the Jupyter frontends, but could e.g. just install `ipykernel` and open notebooks in VSCode.

[1](https://github.com/pyenv/pyenv/) [2](https://python-poetry.org/)

Edit: Moved the links

I do very similarly, except I avoid installing jupyter lab for each project, instead installing `ipykernel` as a dev dependency. I install jupyter lab system-wide with pipx [1], and for each project issue a command like

    pipenv run python -m ipykernel install --user --name=this_directory
Then if I open Jupyter Lab I see "this_directory" as a listed kernel to create a notebook from.

This allows me to manage Jupyter settings and plugins in one place rather than in each env, have multiple project's notebooks open in the same Jupyter Lab instance, etc.


Mamba is a reimplementation of the conda package manager in C++. (quote from README at https://github.com/mamba-org/mamba, linked to in original post)

Funny, Python is too slow for a package manager and most people who can write fast C extensions have left the scene.

So C++ is the natural choice.

Not sure if I agree with you but I was very surprised to find that their Python kernel is also implemented in C++: https://github.com/jupyter-xeus/xeus

> most people who can write fast C extensions have left the scene

Care to elaborate?

Have you looked at any of the Jupyter Docker Stacks images? [1]

I've built some personal Docker images based on these with quite a lot of additional features installed including Ruby, OpenCL and lots of additional Python and R packages and JupyterLab extensions. It's been a bit hit and miss - with extension incompatibilities being a bit of an issue. I sync directories on the host machine with the Docker image and so have the benefit of editing in a full text editor (for non-notebook scripts).

I'm not a Python env expert so I've probably missed a few tricks but all seems to be working well now. Happy to help if you want to try this route and have any issues.

Btw after using JupyterLab for over a year now I'm a huge fan.

[1] https://jupyter-docker-stacks.readthedocs.io/en/latest/

I have an unconventional setup of Jupyterlab, Python dependencies, Swift (through PythonKit package) with Bazel. Surprisingly, the new `rules_python` and `pip_install` support works great at installing packages through pip. This ensures on any machine, I will have a consistent Python runtime (either downloaded or build from the source), as well as the pinned python dependencies when the repo checked out. It also helps because I have a Swift kernel that packaged inside the repo as well.

One thing I haven't figured out is about Jupyterlab's extension system, which previously requires node.js for delivery. It seems 3.0 removed that requirement, so I am hopeful with some tuning I can deliver the plugins in consistent way as well.

Neat! I don't suppose you've got your BUILD and WORKSPACE up somewhere you wouldn't mind sharing?

mamba is (nominally) a drop-in replacement for Conda's CLI and dependency resolver. I've been using it for a couple of months and it really does run an order of magnitude faster for anything that requires dependency resolution. You'll still want to keep ana/miniconda around, but it goes to show how some of Conda's performance woes are entirely self-inflicted.

On linux, I install Anaconda and just launch JupyterLab from Anaconda Navigator.

But you could just install conda and just `conda env export -n base` to export your environment.

I've seen a few projects that have used the JupyterLab UI for other projects as a simple interface - it looks really neat and slick.

I just wondered if anyone has any ideas how you go about this as I've been drawing a blank.

Are you looking for PhosphorJS? https://phosphorjs.github.io/

It's the UI platform Jupyter built to power JupyterLab but is also a general purpose UI framework.

The original author of Phosphor decided to stop working on it, so JupyterLab forked it into Lumino - https://github.com/jupyterlab/lumino

Could you share any examples of what you've seen?

I'm so sorry, I don't remember - I had a demo of an app about 6 months ago which went no where, but, the interface was amazing and they told me they used the Jupyter project's interface.

I tried downloading the source and removing the majority of the bits, but, I just got absolutely nowhere and was hoping there was a library somewhere or a guide that would make this job much easier, however, I'm getting nowhere!

Thanks. Do you remember what was amazing? I've been using a fairly vanilla JupyterLab setup as the basis for a (proprietary) project. I've found the docs generally OK for what I've been doing.

To be honest one of the worst things about JupyterLab I've found is the number of interesting extensions that look great but turn out to be unmaintained or overlap with others. It can be really confusing!

It just worked!

Basically, we build a lot of internal webapps and it's so common that you need to compare x against y, go back and forth between screens...

Most users simply open up multiple tabs and view side by side.

To me, if we could easily have the interface setup so that every object opens in a new "window" with the ability to dock side by side/navigate independently, I just feel it is much neater than the standard admin templates we use on a daily basis.

Truthfully, I've not spent ages looking in to this, but, I did spend a few hours with the Jupyterlabs code and I just got nowhere.

Basically, I would love to just get a page up, the full Jupyter interface, standard menu bar to the left or on top and the ability to just load multiple "Hello World" pages in separate windows.

This one uses JupyterLab as a frontend for CadQueury, a porcedural CAD system.


Thank you that's really interesting. I've been working on a (proprietary) project that uses JupyterLab as a front end - for actuarial / insurance calculations. Happy to share experiences even if I can't share the underlying code (which probably wouldn't be of interest anyway!).

anyone still like classic notebooks better???

Yes! Every now and then I give JupyterLab a go but I keep going back to classic notebook. The only two extensions I need is Vim key bindings and Black formatting. I might checkout version 3 in a few months, but classic has a “feel” that just works for me. I’ve also tried the VSCode abs PyCharm notebooks. The dealbreaker is always how it displays the output of cells. It sucks compared to classic.

Then you might be interested in a new project called JupyterLab Classic, an alternative JupyterLab distribution with the Classic Notebook look and feel: https://github.com/jtpio/jupyterlab-classic

Once the vim and black extensions adopt the new extension distribution system, they should also be compatible with JupyterLab Classic.

I do. The find and replace is better. You can do it across multiple cells. I don't like the drag and drop method of moving cells. Switching between notebooks is faster if you normally use multiple large notebooks at the same time. I am biased against JavaScript heavy UI. I just have had so many bad experiences with single page apps. The only problem I have with notebooks is the mathjax J's seems to cause the browser to hang on opening or running large notebooks. But it was the same in labs, but even worse with the tabs being handled by the UI instead of the browser. Switching back and forth between tabs and having to wait for them to render was annoying. I gave it a good go for a while recently, but the find and replace in classic notebooks is just so good I couldn't make the switch permanent.

Nice ! Native debugger :)

Yes, debugging notebooks is not the best experience so I'm glad there are improvements. I will have a look.

Really hope they can change their notebook editor to monaco.

Applications are open for YC Summer 2021

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact