
Pyodide: Bringing the scientific Python stack to the browser - barryvan
https://hacks.mozilla.org/2019/04/pyodide-bringing-the-scientific-python-stack-to-the-browser/
======
TeMPOraL
Wow, it seems to work really well. After giving it a small test drive, I'm
impressed. Solid work from Mozilla, and a pretty informative article too!

That said, between this, iodide and ObservableHQ, I guess I give up. The
browser _is_ the new OS.

My question now is - how can we make browser environment to be more like Emacs
(bear with me)? My main complaints are:

\- Browser ergonomy absolutely sucks, and there's nothing you can do about it.
You're at the mercy of each service's UI, which usually means you'll be
clicking a lot and maybe sometimes get a keyboard shortcut here and there.
Forget about advanced features, or consistency between sites.

\- You have near-zero control over your computing environment. Between
sandboxing and half the code being obfuscated blob of transpiled JS (the other
half of the code is on the server), you can maybe automate the UI a little
with userscripts and fix it up with userstyles.

\- There's near-zero interoperability. Unless service authors agree on and
implement some APIs, things can't talk to each other. Forget about _making_
them talk. Whatever little wiring you can sometimes do (thanks to services
like Zapier and IFTTT), you never have control over the process, and it always
involves communication through third-party servers, even if all you'd like is
to transfer some local data between browser tabs.

If the browser is the next OS, can we make it suck less than desktop OSes in
terms of ergonomy / productivity? Desktop OSes already suck in this regard
compared to the promise of old Smalltalk and Lisp systems (that's why I live
in Emacs most of the time), so all I see is a downwards trend.

Current state of things is fine for casual use and casual users, but if I'm to
spend 8+ hours a day doing serious work in a web browser, the browser needs to
be better.

~~~
scribu
> If I'm to spend 8+ hours a day doing serious work in a web browser, the
> browser needs to be better.

Agreed! Jupyter notebooks are great, but I miss my Vim keybindings when
editing code in cells.

(Plugins like Vimperator always collide with _some_ native keybindings.)

~~~
hipplec
If you run EIN (Emacs Ipython Notebook module) in emacs with vim keybindings
(via spacemacs or evil mode, or some other distribution/mode) you can get some
pretty great results. Would definitely look into it if you're either already
using emacs or willing to try it out.

~~~
jaster
I tried it at some point, but a minor inconvenience was that it kept adding
superfluous metadata to the cells I was editing.

Since some of the notebooks I work on are collaborative and versioned with
git, this was painful for reading diffs (even more so with the ipynb format)
so I dropped it to my regret.

------
amasad
It's great to see how far WebAssembly and Emscripten has come and this is a
really cool app!

> If you haven’t already tried Pyodide in action, go try it now! (50MB
> download)

I wonder though how ready is it for production usage. At Repl.it, years
ago[1], we moved away from browser-based execution to the cloud because it
excluded many users who don't have the client-side firepower to
download/parse/execute this much JS/WASM.

Repl.it can already run a lot of the examples here [2] but I gotta say the
DOM-integration is pretty neat. We could do interactive stuff on Repl.it using
our new Graphics infrastructure (GFX)[3] but there is always the roundtrip
delay. Here is the same matplotlib example running and streamed down on X11:
[https://repl.it/@amasad/matplotlib](https://repl.it/@amasad/matplotlib)

[1]:
[https://news.ycombinator.com/item?id=16578943](https://news.ycombinator.com/item?id=16578943)

[2]: [https://repl.it/@amasad/pyodide-
example](https://repl.it/@amasad/pyodide-example)

[3]: [https://repl.it/blog/gfx](https://repl.it/blog/gfx)

~~~
azakai
Browser vs server-side is always going to be a tradeoff, yeah. In some cases
you're ok with paying for server time, and then don't need to depend on
clients running your code. But in other cases it's much more cost-effective to
run code on the client.

Note that the browser has gotten a lot better in the years since repl.it moved
away from running code there. In particular wasm parses and executes a lot
faster and takes a lot less memory than JS.

~~~
amasad
We're overdue on toying around and reevaluating wasm. It would be particularly
great if we can make the decision on a per-use-case basis. I.e. fallback to
cloud in underpowered devices or instances where it makes more sense.

------
drej
There is one area where "computing in the browser" misses the mark a little -
browser interfaces (like Jupyter) are often used not just for the convenience
they offer, but also because they serve as thin clients, which provision
vastly better computing resources than you have locally.

So while I do some of my coding in the browser, the code executed is an EC2
far far away, close to my data, with excellent networking etc. There is very
little that a Python stack directly in my browser would offer, to me (!) at
least (your mileage may vary).

~~~
_visgean
Not just that, but a lot of organization use the setup to give good starting
interface to data scientist who might know nothing about how to set up all the
python packages...

~~~
maxnoe
A data scientist wo knows nothing about how to setup the tools he or she uses
daily?

Tell me more.

~~~
_visgean
Well I have some friends doing genetics and their infrastructure is maintained
by IT team. For them the whole programming experience is connecting to jupyter
server running on pre-configured server. I think there are even some online
services offering this kind of configuration for teams.

------
jononor
Incredible undertaking. Looking forward to play with it for simple machine
learning tasks. I think that loading packages directly from PyPI will be a
huge milestone.

------
saboot
Would be curious to see if a version of Cython could be made to work with
this. Much of my data analysis is me passing numpy arrays into C functions
which Cython helps a lot with. So I would be looking for a version of Cython
that would convert that to webassembly.

~~~
mdboom
Cython works for ahead-of-time compilation. (Pandas requires it, for example).
Making it work in the browser would also mean putting a C compiler there,
which people have done. I have no idea how well that would all hold together,
though.

~~~
jononor
Ideally Cython would allow to compile directly to Webassembly (and maybe using
Pyodine converters), without going through C and requiring a C compiler also.

Of course that is a huge job and might very well never happen.

------
VectorLock
I want to believe so bad. Being able to do web development with Python instead
of JavaScript has been something I've dreamed about forever.

~~~
aldoushuxley001
Why not just use Django or Flask? Python web dev is great because you do have
access to so many of these incredible scientific libraries like Pandas, etc.

~~~
_raoulcousins
I don't know anything about web dev, so maybe it's a stupid question, but you
still need javascript for the front end, right? You can't build the front and
back end development completely in python?

~~~
dfinninger
I think they're implying that you use a Python web framework and server-side
rendering.

------
Hendrikto
> It’s also been argued more generally that Python not running in the browser
> represents an existential threat to the language

I do not believe that.

------
jahewson
Is anyone using this yet? What’s your experience been?

~~~
eapartridge
I've used it for a couple small projects in the context of trying Iodide.

It's pretty good for data science—I can't speak to DOM manipulation that much,
I've only done very minimal work with it for that. My two cents: it's bit
rough around the edges—the performance is noticeably worse, and for data
science, with medium-sized datasets it can freeze a tab for 10-15 seconds.
Sometimes you get an odd edge case though where standard Python cannot run
perfectly due to some combination of operations.

That said, overall, I'm really excited by where it's going, and I've always
been able to work around the limitations.

------
outlace
Can this be used to create a desktop application using Electron?

~~~
arduinomancer
That would be an interesting stack if you could write desktop apps using
Python but use the DOM/Browser stuff for the UI layer.

~~~
outlace
Yeah I often run into the need to develop basic UIs for data analysis for
people with limited programming experience. Currently I have to settle for
using Jupyter notebooks with embedded ipywidgets, but it's not still not very
user-friendly. And the desktop app libraries like PyQt5 are overkill for what
I need, and most of the visualization libraries I use are designed for jupyter
notebooks/web e.g. Bokeh.

If I could just port my jupyter notebook+widgets+plots into an Electron window
and ship it to people that'd be awesome.

~~~
geophph
Maybe check out Dash by Plotly. It's helped me address that exact situation as
well.

------
snthpy
Looks really nice!

I prefer the look of the JSMD format to the Jupyter notebook format and hope
that is something that might get integrated into JupyterHub as well.

Question: is there some documentation for deploying your own Iodide server
behind a firewall? The original Iodide post mentions this is possible.

------
alexgmcm
It's a cool project technically, but I don't understand the use case?

I mean surely everyone can just have real python/jupyterlab - if you run it in
Docker it's easy to handle the dependencies etc.

Am I missing something?

~~~
callahad
I gave a keynote at PyCon last year that touched on this:
[https://www.youtube.com/watch?v=ITksU31c1WY#t=32](https://www.youtube.com/watch?v=ITksU31c1WY#t=32)

For me, the most important aspect of Pyodide is that it allows the scientific
Python stack to go everywhere the Web goes. And the Web goes _everywhere._

That means I can share a link to a Pyodide notebook, and the recipient gets a
fully interactive experience, no installation required. And that's _really_
important for the generation that's getting their start in computing through
classroom iPads, Chromebooks, etc. which do not easily allow for "real"
Python.

~~~
Vinnl
As someone not familiar with any of this: is that not exactly what Jupyter Lab
does? I understood that the main difference between Pyodide and that is that
Pyodide allows you to also use e.g. D3 for rendering?

~~~
callahad
A big difference is _where_ Python runs. With Pyodide, everything runs
entirely in your browser, and the server side can be completely static. With
Jupyter, you need a separate copy of Python running on a server somewhere to
actually perform any computation.

~~~
amelius
Wouldn't it be more straightforward then to compile Python to WASM, and
combine the resulting binary with Jupyter so everything can run in the
browser?

~~~
stult
Compiling Python to WASM was far from straightforward. That's essentially all
Pyodide is, and why it is impressive. Presumably no one has taken the time to
combine it with Jupyter since it is so brand spanking new.

------
jdpigeon
I'm looking forward to try out Pyodide in Electron so that I can make use of
all the great Python scientific and signal processing libraries for data
processing.

~~~
mlevental
lol what a crazy rubegoldberg machine - like the person beneath asks: why not
just use "native" python

~~~
stult
PWAs, running in a sandboxed/secured environment where installing native
python isn't an option, bolting existing python code bases onto existing
electron code bases. There are plenty of use cases. Just because you haven't
encountered one doesn't mean there aren't legitimate applications. It's
hard/impossible to judge someone's tech stack without understand the
constraints they are operating under.

~~~
codesushi42
Exactly. Otherwise you'd need some other way to communicate between Electron
and Python. This is possible, you could use ZeroMQ. But it would be a huge
pain.

This would easily make sense where you want to execute your app's logic in
Python, but want to keep the UI in HTML and JS.

~~~
stult
Definitely. And a lot of the motivation for that type of approach is driven by
team skill portfolios rather than raw technical merit. If you have a data
science team specialized in python and a front end team specialized in JS
(which I think we can agree describes the overwhelming majority of DS and FE
teams), there's a really strong organizational motivation to bolt python and
JS together in flexible ways that match the available deployment
infrastructure, which, like it or not, includes electron.

------
ngcc_hk
Can it run cuda? Otherwise not that practical.

Minor interest. Can this approach run j/k or more important lisp?

~~~
mkl
_Most_ practical python programs do not use CUDA, even scientific ones.

~~~
BeetleB
Says the mkl user.

(Sorry, couldn't resist).

