
Jupyter, Mathematica, and the Future of the Research Paper - bachmeier
https://paulromer.net/jupyter-mathematica-and-the-future-of-the-research-paper/
======
dahart
> The tie-breaker is social, not technical.

The tie-breaker is financial. Jupyter is winning because it's free, not
because it's social. It becomes social because of widespread adoption, and it
get's widespread adoption because it's free.

I _love_ Jupyter, love love love. But there's a lot of hyperbole and opinion
here. Mathematica is just a for-profit business, it's that simple. And it
wouldn't be fair to deny the example that Mathematica, Maple & Matlab have set
for free solutions like Jupyter.

There's nothing dishonest about a for-profit business. And if Mathematica
wants to keep PDF export for themselves, so be it, that's their right. What's
dishonest is expecting technical software and service for free and calling
people names like vandals if you don't get it. Just celebrate Jupyter and
enjoy that people are doing great work you get to use without paying. I don't
love Mathematica or it's founder, but there's no real need to impugn
Mathematica in order to make this point.

~~~
gaius
_The tie-breaker is financial. Jupyter is winning because it 's free_

This is it really

Back in the 90s I was using a program called MathCAD, it provided a “notebook”
interface by running as a plugin to Word 6. In terms of general usability and
experience, 20-years-ago blows away modern-day Jupyter and it’s silly “cells”
interface, which it does not because it’s better but because it’s trying to
force itself into a web browser. I haven’t used MathCAD since but I bet in
2018 it’s _amazing_.

I think few people who have used the commercial tools think Jupyter is better.
But the commercial tools are soooooooo expensive...

~~~
sanderjd
I used MathCAD in physics and chemistry classes in college. I frequently
wonder what happened to it, because I share your perception that while Jupyter
is very nifty, it is hamstrung by the limitations of the browser environment.

I'm waiting impatiently for the coming revolution of non-web collaborative
internet-connected rich client applications.

~~~
tincholio
As a die-hard Emacs user, I think that org-mode with it's org-babel
capabilities blows Jupyter out of the water, and it produces much better,
readable output. You can use it with pretty much any language, and combine
several languages in a single document without any issues. Besides all this,
it's just a plain text format, and you can extract all the code into proper
source files for later offline use, too ('tangling', in literate programming
parlance). Jupyter uses a rather obtuse json format which is not practical to
work on directly.

Of course, it's not web-based and it requires some basic knowledge of Emacs,
but functionality-wise is so much better that it ends up being frustrating to
use Jupyter when collaborating with other people. There are some workarounds
(EIN, and ob-ipython modes help, but it's not quite the same)

~~~
todd8
I share your opinion about the superiority of org-mode as a means to assemble
a document out of various pieces. It's not hard to show nicely formatted code,
the code's output, the math behind the code and perhaps a graph of the results
all in a LaTeX/PDF document and or HTML. All of this can be done with a flat
text file that is easy to save as a part of a project under source control.

What I wonder is why are Jupyter notebooks so popular? I've tried them, but
perhaps I don't understand the workflow in the way people use them. Are they
intended to be works in progress where one explores ideas and looks at
results? This is how I've usually used them and Mathematica's notebooks as
well.

For me though, my Jupyter or Mathematica notebooks end up with lots of false
steps and dead-ends. To prepare a presentable form of the results I have to go
back and edit the notebook to such a degree that it doesn't feel like working
in a notebook as much as it is working in a complex document with a really
weak set of editing tools. To be fair, I haven't used Jupyter or Mathematica
for a couple of years and things may have changed. Maybe HN readers can help
be understand how to more effectively use these tools.

~~~
dahart
> What I wonder is why are Jupyter notebooks so popular?

How easy is it to share a Jupyter notebook with someone else, versus sharing
an org-mode notebook? Think about the recipients rather than how hard it is
for you. Assume the person you share with needs to be able to re-run your
notebook. Assume the person you share with isn't using the same operating
system that you are. Be honest and think about all the steps involved.

With Jupyter, I send a single install link to Anaconda. With Emacs + org mode
+ python + numpy + scipy + matplotlib, I might not even know where to begin if
the recipient is using Windows or Mac. The recipient needs to install emacs,
and they need to know how to use Emacs. They have to know how to install
python packages (and possibly a package manager), and ideally be able to use
virtualenv too.

If I need different libraries than what Anaconda comes with, I can put the
commands in the notebook, and I won't mess up the recipient's system version
of Python. With org-mode, they'd need to figure it out on their own.

If I choose to use a different language than Python in my org-node notebook,
the recipient will have to install that programming language and environment
themselves before they can use your notebook.

It's the simplicity and bundling of dependencies that make Jupyter popular.
All the flexibility and power you get with org mode comes at a high price.

~~~
tincholio
> How easy is it to share a Jupyter notebook with someone else, versus sharing
> an org-mode notebook?

Well, now you're talking about a whole environment. Use docker, or VMs or
whatever. There's plenty of options for doing this (we actually use Docker for
Jupyter, to make sure we work off a consistent install, so I guess sharing can
be non-trivial if you walk a bit off the beaten path, anyway)

> If I choose to use a different language than Python in my org-node notebook,
> the recipient will have to install that programming language and environment
> themselves before they can use your notebook.

Sure, and if you need to use another language from your Jupyter notebook, you
can't. I don't get your point. Yes, if you need to use a tool, you need to
install it. It requires a minimum of computer literacy.

~~~
dahart
Yes, Anaconda + Jupyter is an environment and org mode is not. That’s a big
reason Jupyter is so popular.

> I don’t get your point.

My point was Jupyter inflicts far less technical burden and debt on other
people. There’s a huge downside to custom environments. If that’s not obvious,
that explains why it’s hard to understand jupyter’s popularity.

> it requires a minimum of computer literacy

I’m sure you didn’t mean it that way, but this borders on white tower
programmer centric dismissiveness.

Personally I know how to install all the things an org mode notebook might
require, but I don’t _want_ to. It takes too much of my time and energy.
Expecting other people to adopt your choice of tools and environment by
arguing that your tools are basic literacy is presumptuous. Use org mode, I’m
wildly in favor of people using tools that they like, and you like it. Just
don’t claim it’s objectively better on all axes when it’s not.

~~~
tincholio
> Yes, Anaconda + Jupyter is an environment and org mode is not. That’s a big
> reason Jupyter is so popular.

You're again comparing apples to oranges. If you want to use Jupyter, you need
to install a whole lot of stuff (easier to do if you're using Anaconda,
agreed). It's "not simple" enough that there are public docker images for it.
My point is that when tooling is an issue, you can do the exact same thing
with an org-based setup.

>My point was Jupyter inflicts far less technical burden and debt on other
people. There’s a huge downside to custom environments. If that’s not obvious,
that explains why it’s hard to understand jupyter’s popularity.

I agree that there's difficulty in achieving reproducible environments, and
that's why there's tools that handle that for you. I think that much of
Jupyter's popularity comes from the fact that it's easily available, and based
on Python, which let's non-programmers get started moderately quickly. I just
don't think that it's the best choice. This is not to shit on Jupyter, I've
mentioned a bunch of advantages of using org (which by the way, you haven't
really addressed, other than focusing on the setup aspects)

> I’m sure you didn’t mean it that way, but this borders on white tower
> programmer centric dismissiveness.

It's not, really. My point is that when you're doing presumably advanced
analysis on presumably complex data, you (should!) certainly have the capacity
to deal with tooling.

------
FractalLP
Long time user of Python here and recent user of Mathematica.

Some observations I have are that they're both great. Python is a nice open
source scripting language, but getting libraries to work can sometimes be a
pain. Mathematica is basically install this and everything is included. The
Mathematica documentation is amazing and it is really simple how to do most
things. The whole iPhone "there is an app for that" is equivalent to "there is
a function for that".

Graph Theory works flawlessly in Mathematica. In Python, there is a module to
Graphviz. Let me know if Python has something new though. There are a lot of
other examples. Mathematica's Import[] function can read over 150 different
file types including: CSV,.XLS, genetic encoding files, optimization
files....whatever. It is usually far easier and more consistent than finding a
corresponding Python library and struggling with the install and minimal
documentation. Let me be clear that Python is awesome and rocks and i think
Jupyter is moving it in the right direction. I just feel like many dismiss
Mathematica as something that does Calculus homework rather than what it is
today which is a massive 20 million LOC conglomeration of C & Java & Wolfram
language that does everything from Statistics, Machine Learning,
Visualization, BlockChain, 3D printing, NodeGraphs, data sets and
analysis...etc in a single consistent package. It is expensive and proprietary
and certainly has its own faults, but a lot of that cash is funneled back into
a great product.

~~~
askvictor
While I totally hear you regarding the pain of python modules (particularly on
Windows), the point of python 'distributions' like anaconda and canopy is to
bring the kitchen sink along, kind of like mathematica.

The problem with Mathematica from a science point of view is that, being
closed source, means you can't independently ensure the calculations are
happening correctly. To be replicable, science involving data needs to use
open source tools.

~~~
gaius
_The problem with Mathematica from a science point of view is that, being
closed source, means you can 't independently ensure the calculations are
happening correctly_

Have there been any high profile failures root-caused to Mathematica (or
MATLAB or any similar product) getting its sums wrong? I can find any news
stories etc. Plenty of serious calculations were and are done on “closed
source” HP and TI calculators too. Every serious scientific instrument with
its own data capture uses a binary blob too somewhere, so if that’s a problem
for you then you can’t even trust the raw data!

And even if you have all of the code - you still need to worry if the
proprietary, closed FPU is working “correctly”.

This sounds very much like a post-hoc justification for “its free as in beer”.
Do you think Wikipedia is more trustworthy than real references too? What
about blogs?

~~~
PopePompous
Science publication is moving (very, very, very slowly) towards a model where
instead of a final, polished traditional paper, the raw data along with the
software tools and interpretation is published. In principle this should allow
readers to completely understand and reproduce the processing of the raw data,
rather than reading a few paragraphs summarizing the processing done by the
authors. Using a closed source tool for processing the data limits how deeply
a reader can delve into the processing that the authors did, because the
functions in the proprietary package are black boxes. Jupyter has no black
boxes.

~~~
gaius
_In principle this should allow readers to completely understand and reproduce
the processing of the raw data, rather than reading a few paragraphs
summarizing the processing done by the authors. Using a closed source tool_

But consider [http://www.bbc.co.uk/news/science-
environment-39054778](http://www.bbc.co.uk/news/science-environment-39054778)

"Science is facing a "reproducibility crisis" where more than two-thirds of
researchers have tried and failed to reproduce another scientist's
experiments"

I don't think that can be handwaved away as "OMG closed source software!".
Especially since all the scientists in a given field will have access to the
same software anyway. Give them open source and the issue will persist, and we
both know it because the root cause isn't anything to do with the license of
the software

~~~
nmca
Open source software, in general, tends to make things more reproducible.
Sure, software licencing might not be the singular root cause, but why does
that suggest we shouldn't capitalise on the improvements available?

~~~
gaius
_Open source software, in general, tends to make things more reproducible_

Citation very much needed for that. Because you can very easily find that 6
months or a year later you update your dependencies and everything is now
broken. I recently came back to a Python project after a year, updated my
packages then realized: I simply cannot be bothered to unpick the mess that
resulted just to add one trivial feature. Whereas the poster child for
backwards compatibility is closed-source and proprietary.

Science is not reproducible because there are no incentives for it to be so,
despite everyone paying lip service to it. It's extra work and helps those who
are competing with you for grants, after all. That's a social problem, not a
software one. The software is irrelevant.

~~~
takluyver
Open source software is important for reproducibility for a couple of reasons.
Firstly, if you record that you've done your analysis with Python 3.6.3 and
Numpy 1.14.2, and it later breaks on some newer version, it's relatively easy
to get the same versions you were using. Commercial software vendors are
usually not keen on you downloading and running a version of their product
which was superseded four years ago.

Secondly, of course, open source means that if you're not sure why two
versions/functions/libraries are giving you different answers, you can go and
find out. I accept that a lot of people may not have time for that, but I
don't think you can fix that problem unless it's possible to dig down and
follow the working.

Finally, 'reproducible by anyone with a computer' is a lot better than
'reproducible by people who buy a license for the tool I used to do it'.

~~~
gaius
_f you record that you 've done your analysis with Python 3.6.3 and Numpy
1.14.2, and it later breaks on some newer version, it's relatively easy to get
the same versions you were using_

Better record which compiler you used too, and what flags, and every version
of every library and everything else. It’s not as simple as you make out and
it’s far from guaranteed that all those packages will still be available or
compile on your OS.

 _Commercial software vendors are usually not keen on you downloading and
running a version of their product which was superseded four years ago._

I guess you must not deal with vendors much because generally they are fine
with this. It’s part of the support agreement usually, just another service.
Getting an “obsolete” version for whatever reason has never been a problem for
me.

By “anyone with a computer” you mean “anyone who can exactly reproduce my
configuration which I don’t even know myself for certain”

------
cs702
This is spot-on:

"Membership in an open source community is like membership in the community of
science. There is a straightforward process for finding a true answer to any
question. People disagree in public conversations. They must explain clearly
and listen to those who response with equal clarity. Members of the community
pay more attention to those who have been right in the past, and to those who
enhance their reputation for integrity by admitting in public when they are
wrong. They shun those who mislead. There is no court of final appeal. The
only recourse is to the facts.

It’s a messy process but it works, the only one in all of human history that
ever has. No other has ever achieved consensus at scale without recourse to
coercion.

In science, anyone can experiment. In open source, anyone can access the facts
of the code. Linus Torvalds may supervise a hierarchy that decides what goes
into the Linux kernel, but anyone can see what’s there. Because the
communities of science and open source accept facts as the ultimate source of
truth and use the same public system for resolving disagreements about the
facts, they foster the same norms of trust grounded in individual integrity."

The entire blog post is worth a read.

~~~
gaius
_Membership in an open source community is like membership in the community of
science. There is a straightforward process for finding a true answer to any
question_

Oh please. Dare to ask what is the best of anything and prepare for an epic
flame war.

~~~
ChrisRackauckas
Well, I asked the author for the data that led him to believe that the Julia
community is "monopolistic", and he got mad and blocked me instead of linking
to anything...

[https://twitter.com/ChrisRackauckas/status/98552939470474035...](https://twitter.com/ChrisRackauckas/status/985529394704740358)

So your remark is pretty spot on.

~~~
littlehood
You asked for that block in the second question. You sound very entitled to
answers.

~~~
ChrisRackauckas
I am just curious what led him to his conclusions. He made a very nasty
comment about a group of people and gives no justification for why. I cannot
seem to find out why when Googling either. Is it not okay to ask someone how
they came to their conclusions?

~~~
littlehood
You could've asked politely, but it looks you missed your chance.

------
yaroslavvb
I've been using Mathematica since 1995 and Jupyter/colab for 5+ years. Most
recently I've been using them both in parallel. While Jupyter is probably the
future in terms of mass adoption, there are still some areas where Jupyter is
lagging.

1\. Mathematica has an easy way of sharing notebook. I just run "deploy"
command which turns notebook into publicly accessible webpage, hosted by
wolfram, here's an example -- [https://www.wolframcloud.com/objects/user-
eac9ee2d-7714-42da...](https://www.wolframcloud.com/objects/user-
eac9ee2d-7714-42da-8f84-bec1603944d5/treeDecomposition_17-11-27-18-57-24.nb)

2\. Mathematica has more active community. Mathematica-specific questions are
likely to be answered within an hour by experts on
[https://mathematica.stackexchange.com/](https://mathematica.stackexchange.com/)

3\. Mathematica has better tools for simple interactivity. I like to throw in
"Manipulate" for a simple graph with a draggable constant, or go to
[http://demonstrations.wolfram.com/index.php](http://demonstrations.wolfram.com/index.php)
for an idea for more complicated demonstration to use in a presentation

4\. Mathematica has more options for advanced visualization, and interfaces
are more uniform since graph drawing, 3D drawing, and other kinds of
visualizations are developed within a single system. Some examples
[https://www.wolfram.com/language/11/new-visualization-
domain...](https://www.wolfram.com/language/11/new-visualization-
domains/?product=mathematica)

~~~
carreau
Thanks for your feedback, Mathematica has indeed millions of $ to provide more
features and advertise them, and Jupyter have only a few full time devs that
probably do not advertise enough its features:

1) Binder makes that a git push away
[https://mybinder.org/](https://mybinder.org/) Want to check the discovery of
gravitational waves ? Go ahead ! [https://github.com/minrk/ligo-
binder](https://github.com/minrk/ligo-binder) You know the nice thing ? it
does not require you to opt-in, as long as a repo is public you an run it. So
you don't need have to deploy, or know it exists. we are _already_ doing that
for you.

2) Jupyter is "Just" the frontend. StackOverflow have matpltlolib, numpy,
sympy, .. tags. We don't the subdomain (yet), and I actually prefer to have
tags to have better searching :-)

3) Sure it's called ipywidgets
([https://ipywidgets.readthedocs.io/en/latest/](https://ipywidgets.readthedocs.io/en/latest/)),
that's the tech. From ipywidgets import interact, and @interact as decorator
on your function... that's it.

4) For convenience Library that use ipywidgets for 3D see
[https://ipyvolume.readthedocs.io/en/latest/animation.html](https://ipyvolume.readthedocs.io/en/latest/animation.html)
(Hey it also support VR !) See
[https://www.youtube.com/watch?v=nZ3HQpSXn2U](https://www.youtube.com/watch?v=nZ3HQpSXn2U)
that will blow your mind.

We'll try to be better at advertising our features !

~~~
smortaz
MyBinder is great. As well as cocalc and sagemath. There's also
[https://notebooks.azure.com](https://notebooks.azure.com) which is a free
hosting of Jupyter notebooks. You get a linux/docker container w a Terminal,
Anaconda, etc.

Check out jakevp's book for example:

[https://notebooks.azure.com/jakevdp/libraries/PythonDataScie...](https://notebooks.azure.com/jakevdp/libraries/PythonDataScienceHandbook)

Clone to run.

Also try Jupyter Lab (experimental) - closer to an IDE than plain notebooks.
right click on a Library (repo) and select Open in Jupyter Lab.

[disclaimer - our team's offering]

~~~
applecrazy
There’s also Google Colaboratory

------
iguy
Right, Jupyter is nice to have, but this is really over the top nonsense:

 _Jupyter encourages individual integrity; Mathematica lets individuals hide
behind corporate evasion_

I have no idea what he's talking about re PDF export either. I print to PDF
all the time, to email people a static document to look at, etc. It works just
fine. (Whether you can you make book-quality formatted text easily, I've no
idea, never been tempted to try.)

~~~
rexpress
I suspect that the OP used the "Print..." command in the File menu, and
selected PDF as the printer option. ISTR that this can sometimes result in
poor quality results as presumably it is relying on an external PDF engine to
render the notebook.

Whenever I've used the "Save As..." command, choosing PDF as the target, I've
also only had good quality output.

~~~
ChrisRackauckas
There was a bug in Mathematica 10.0.3 that messed up saving parts to PDFs
though. I wonder if he was on the release where this was broken:

[https://mathematica.stackexchange.com/questions/68893/save-a...](https://mathematica.stackexchange.com/questions/68893/save-
as-pdf-broken-in-version-10-0-2)

------
sago
Jupyter is an amazing and useful piece of software. I agree that its openness
is important, that its flexibility in producing content is excellent, and that
it deserves to be the current hotness. But I'm afraid

> Now, Jupyter is the unambiguous technical leader.

is pure fantasy, imho. SymPy is still two decades behind Mathematica in large
swathes of symbolic computation.

It may be that, for the things that the author wanted to do, the Python
libraries were a good fit (it seems he was working on NLP), but overall, I
just don't see it.

------
askvictor
I'm surprised there's no mention or discussion of the importance of open-
source tooling for replicable science. Without seeing and reviewing the
source, how can you tell that a particular calculation is right? Also, relying
on costly tools such as Mathematica cuts off a sizable amount of the
population from being able to replicate or play with your findings on cost
grounds alone.

~~~
falkod
Long-term Mathematica user (physicist) here: I don't think the use of open
source software would make most science -- maybe that does not apply to
cs/datascience -- more replicable. Usually that takes an expert in the field.
And usually these experts are employed at universities where Mathematica
licenses are not the prime cost factor. That said, I am all for open source
software. Although I would argue that probably trustable scientific results do
not rely on the inner workings of e.g. Mathematica anyway, but use Mathematica
as vehicle for say linear algebra or symbolic manipulation etc. While the
inner workings of Mathematica may not be open source, in principle the
relevant algorithms are not propriertary but usually well-known mathematical
results and as such at least in principle easily reproducible outside of the
ecosystem.

~~~
jononor
Open source brings with it a strong _culture_ of publishing code openly. It
seems that the amount of public Jupyter notebooks is already higher than that
of Mathematica or similar, despite those tools having a 20 year lead?

------
hpcjoe
Responders have been making the argument that Jupyter is beating Mathematica
because of financial or social issues. I'd like to posit a different
interpretation, which could be construed to encapsulate these reasons, as well
as additional other factors.

Jupyter has a lower friction to adoption and usage than Mathematica, for a
definition of friction which encompasses ease of acquisition and sharing. I
include economic considerations in the ease of acquisition and sharing. Lack
of proprietary walled garden lock-in/lock-out factors in as well.

People are also likely considering the longer term scenario, whereby data,
model, and information interchange has been hindered by proprietary formats
(the "wall" in the walled garden) and lack of complete information on how to
get information in and out. Which is what the OP was complaining about, as
they were not able to easily construct a publication quality
preprint/submission from one, but could do it easily from the other.

Some of these sources of friction are effectively "own-goals", that is, you
increase friction in such a way as to prevent something that people need to
do, to be effectively impossible. Or you hide it. Or disable certain groups
from using that functionality.

Then the question is balancing the longevity of the format, the proprietary
value against alternatives. Increasingly, people are less interested in this
friction for a number of critical systems.

I am looking at this from the perspective of someone who has a few 10's of MB
of data/writings on 25-30 year old 3.5 inch and 5.25 inch floppies. These are
in formats for which I may not have an ability to extract the data/information
without some significant effort.

The formats that have survived well for me over the last 30 years have been
either open, or readable/writeable with open tools. The closed ones, not so
much luck with.

------
limeblack
FYI there is a not as complete open source implementation of Mathematica
called mathics[1][2]. In fact it is also Python based just like Jupyter(I
don't think this is a coincidence).

[1]: [http://mathics.org](http://mathics.org)

[2]: [http://mathics.net](http://mathics.net)

------
ninguem2
He should look into Sage.

[http://www.sagemath.org/](http://www.sagemath.org/)

------
lopmotr
Open source is only great when it exists. For finite element analysis, there
are only two generally useful open source products and neither of them has a
remotely modern or easy UI. For $10,000 or so, you can get a proprietary one
that's fast to use and doesn't have you hitting a brick wall when you find
there's some key feature it can't do.

UI is a major failure of open source - it can hardly ever achieve it, at least
not well. Most of the popular open source programs have no UI at all.

------
forapurpose
> Python libraries let me replicate everything I wanted to do with
> Mathematica: Matplotlib for graphics, SymPy for symbolic math, NumPy and
> SciPy for numerical calculations

Are the Python libraries precise enough for professional mathematicians? And
do they deal with mathematical 'edge cases', a variety of inputs (formats,
notations, etc.), etc.?

On one hand, I could say 'the author uses them therefore they must be
sufficiant'. On the other, I've seen plenty of cases where the professionals
were not careful about the tools they use (e.g, spreadsheets running critical,
large-scale financial operations).

~~~
mlevental
in particular I'm curious if sympy is really as good as Mathematica. I haven't
used Mathematica since doing physics hw as an undergrad but it's symbolic
manipulation was amazing most of the time

~~~
ChrisRackauckas
It's not, but for most people it may not matter. Mathematica seems to have a
much larger set of integrals, differential equations, special functions, etc.
that it can recognize. So as much as I dislike the language itself, I do keep
Mathematica installed because in many cases SymPy cannot handle the
transformation.

On the other hand, open source tools are catching up. Part of the SymPy
project is SymEngine which is a reconstruction in C++. SymEngine is not as
feature-filled as SymPy yet, but it flies. It's much faster than SymPy, and
also Mathematica. SymEngine.jl works very well since by using the symbolic
types in side of Julia you get both the speed (via function specialization)
while getting a lot of free features through generic code. For example, Julia
code is generic over number types, so if you call inv on a matrix of SymEngine
symbols the built-in Base version will compile a fast version for SymEngine
symbols and use it. So you have all of Julia's Base available along with Julia
packages (yeah, you can put them in a neural net if you wanted to). So
SymEngine.jl is the only thing that works when I need that speed, but I do
have to keep Mathematica installed for its special handling of specific
equations.

And I hate to say it but... the Mathematica notebook is really pretty. You put
the exponents up, the fractions on top of each other...

~~~
fdej
> On the other hand, open source tools are catching up.

SymPy is doing nicely, but it's decades behind Mathematica when it comes to
symbolic computation.

By the way, if you want to invert a symbolic matrix, chances are that generic
Gaussian elimination working with symbolic expressions isn't the best
algorithm (it might even give wrong answers if zero testing is done
incorrectly), and compiled code isn't going to change that. The state of the
art in symbolic linear algebra uses specialized algorithms like evaluation-
interpolation, modular computation, and all kinds of low level optimizations.

~~~
ChrisRackauckas
It was a bunch of small matrices so it worked out well, but that's good to
know. For future reference, what library implements these symbolic linalg
routines? When I was looking around I could only find LinBox
([http://www.linalg.org/](http://www.linalg.org/)) but the barrier to entry
was a little high for me to dig in.

~~~
fdej
In Julia, there is Nemo.jl for high performance exact and symbolic linear
algebra. At least that is the goal; it doesn't have the best algorithms yet in
all cases, but it has some of them (and more are being developed). You can see
some examples here:
[http://nemocas.org/benchmarks.html](http://nemocas.org/benchmarks.html)

~~~
ChrisRackauckas
I really like that library for other reasons, but I couldn't find
documentation for anything symbolic. Could you point to the right places? Feel
free to follow up on other forums since this is getting pretty off topic.

~~~
fdej
You have to work with multivariate polynomials (possibly over number fields).
If you have symbolic expressions in another format, you would have to manually
convert them to that representation first, possibly after introducing extra
variables for non-algebraic constants (which may or may not be a good idea
depending on the specific circumstances). Having that kind of rewriting done
automatically would certainly be a good thing!

------
arca_vorago
I have chosen the emacs org mode system over Jupyter, but I still like Jupyter
regardless. The real tragedy is how dependent people have become on
proprietary stacks like Mathematica.

~~~
hatmatrix
org-mode + babel is excellent. But again the social aspect necessary for
adoption is much less developed than Jupyter (the emacs community is very
social, but small).

In that Jupyter files are just json files, I hope that there it will be easier
to switch between the two in the future. Like [1], [2], and [3].

[1] [https://github.com/gregsexton/ob-
ipython](https://github.com/gregsexton/ob-ipython)

[2] [https://github.com/jkitchin/ox-ipynb](https://github.com/jkitchin/ox-
ipynb)

[3] [https://github.com/millejoh/emacs-ipython-
notebook](https://github.com/millejoh/emacs-ipython-notebook)

------
mfe5003
I learned and was fluent with Mathematica early and learned python later. I
still run to Mathematica for doing symbolic analysis because there is
basically no impedement between my ideas and the keyboard when I am solving
that type of problem. I've moved all my numerical analysis to the scipy system
since it is a more natural language for those types of problems.

------
ChrisRackauckas
>Which reminds me. If you are a Julia enthusiast, how do you suppose the
investors in this new language plan to make their big score?

This is a weird jab at Julia. Open source software is woefully underfunded.
Julia Computing was founded in the wake of Heartbleed where people learned
that open source needs some kind of funding to keep developers alive (example
article: [https://arstechnica.com/information-
technology/2014/04/tech-...](https://arstechnica.com/information-
technology/2014/04/tech-giants-chastened-by-heartbleed-finally-agree-to-fund-
openssl/) ). Coming from academic backgrounds, the core contributors really
had two options if they wanted to devote full time to Julia: either everyone
gets an academic job while working on Julia instead of papers (lol), or band
together to get R&D funding and use that to fund a life of open source
development. They did the latter.

It's quite silly to even imply there's something nefarious that can go on
here. Their main product is the language. They can't sabotage that without
sabotaging themselves. They may have some priorities swayed, just like how any
other individual who's working on open source is doing it for their own
reasons. For example, IBM funded them to add PowerPC support, and what do you
know Julia works on PowerPC. Is that so awful? With this funding model, what
ends up happening is you have a large group of people who dedicate their lives
to developing open sourced code for automatic GPU compilation, machine
learning libraries, etc. along with compiler support for optimizing scientific
computing. Because of this (and other reasons), Julia ends up having a much
stronger governance which is one reason why its development ends up being more
active. And this activity in turn makes its project more democratic than
projects like CPython or Jupyter which have been larger projects for a longer
time, but with less contributors (Julia's 686 vs CPython's 524 vs Jupyter's
330).

And most of the Julia contributors aren't even part of that company! Many are
academics. A lot of the funding is through NumFOCUS, a non-profit which also
helps projects like Jupyter, matplotlib, etc. which the author is for! (And
they are great projects as well!)

So while I am happy that the author is pro open source, I think it's necessary
to point out that this open source outsider view is both wrong and dangerous.
Saying that you love the purity and despise anyone who gets to make a living
from it is harmful! Open source is a labor of love, but it has also destroyed
many careers. I think society has this view that open source (mathematical)
projects are "funded" by academic careers, but even creators of popular
projects like SageMath have publicly noted that open source is harmful to
academic success ([https://escapethetower.wordpress.com/2016/06/13/creator-
of-s...](https://escapethetower.wordpress.com/2016/06/13/creator-of-sage-math-
why-im-leaving-academia/)).

Instead of being against funding open source contributors, I would like to see
the author promote funding for open source. Paul Romer is a leading economist.
He has the power to proclaim that open source matters for academic careers and
push for it to be put on equal footing with papers for grant applications in
his field. People like him should be advocating for jobs dedicated to open
source development, not scoffing at the supposed impurity of someone being
paid to develop a public good. Someone at the top of the academic hierarchy
should start a change and make the development of public tools as valued as
the development of (non-public) publications.

~~~
jhbadger
Also the idea of companies making money off of open source isn't new or
nefarious and often helps people who aren't even their customer. Red Hat is
the classic example, and many organizations use CentOS, a distribution based
off of Red Hat's distro without paying Red Hat anything. And in scientific
computing there's RStudio, which makes a great open source IDE for R besides
offering prducts and services for sale.

~~~
ChrisRackauckas
Wow, I asked for the data that led him to his conclusion about "the monopoly"
and he blocked me. So much for open science...

[https://twitter.com/paulmromer/status/985529525491654657](https://twitter.com/paulmromer/status/985529525491654657)

~~~
improbable22
I just realised who this blog post is by, it's _the_ Paul Romer. Apparently
gravitas is important at the World Bank, because clearly he's run the tank
dry.

------
rev
Dan Toomey's _Learning Jupyter_ is free today on
[https://www.packtpub.com/packt/offers/free-
learning](https://www.packtpub.com/packt/offers/free-learning) btw.

------
jonnycomputer
R Studio Notebooks are pretty good too; I like that, by default, there is an
interactive console connected to the same kernel in addition to the notebook.
This allows me to use the console to interactively probe my data, or try out
something, and then record a more finished product in the notebook itself. I
think this can be done in Juypter ([http://jupyter-
notebook.readthedocs.io/en/latest/examples/No...](http://jupyter-
notebook.readthedocs.io/en/latest/examples/Notebook/Connecting%20with%20the%20Qt%20Console.html)),
but, not out of the box.

~~~
heisenzombie
In JupyterLab
([http://jupyterlab.readthedocs.io/en/stable/](http://jupyterlab.readthedocs.io/en/stable/)),
this is built-in. Just right click a notebook and choose "New Console for
Notebook".

JupyterLab is in beta and is intended to replace the current Jupyter front-
end.

~~~
jonnycomputer
I tried JupyterLab out. Definitely an upgrade.

------
wodenokoto
Since the author is talking about using Jupyter for research papers, how do
you do basic things, like bibliographies, naming tables and referencing them
later?

I have seen table of content, but those have been generated by a big block of
javascript.

~~~
llamaz
You can export to LaTeX and hack the "template" (in the terminology of
Jupyter) that defines how this conversion occurs.

I used this as a starting point and modified it for my own purposes:
[http://blog.juliusschulz.de/blog/ultimate-ipython-
notebook](http://blog.juliusschulz.de/blog/ultimate-ipython-notebook)

------
EGreg
It always goes like this.

The initial solutions may be proprietary, and financed by investors. They have
a business model so of course they don’t give everything away for free.

With time, enough people get together to build an open source alternative. And
then like a snowball it eclipses everything proprietary that went before it.

What would a world look like that didn’t apply Capitalism to ideas?

One where companies couldn’t sue one another for Intellectual Property
infringements. Like Waymo suing Uber.

One where self driving cars can incorporate improvements made by any other
self driving car instead of putting people at risk reinventing the wheel.

Where the long tail of drug research leads to something.

Why would people release their findings? Because if they don’t, others will.
And then they won’t get that small measure of input and control and attach
their name to it. Jonas Salk is an exception in the biomedical field. Albert
Einstein is the norm in physics.

The alternatives to Capitalism do not have to be Socialism. They can be
SCIENCE. OPEN SOURCE. WIKI.

Collaboration instead of Competition.

I would like to see the same in web browser engines. WebKit instead of IE. And
so on. When that happens, we all win.

Yes free software is good. Software, like knowledge, does not have to be
scarce.

~~~
TeMPOraL
> _It always goes like this._

> _The initial solutions may be proprietary, and financed by investors. They
> have a business model so of course they don’t give everything away for
> free._

> _With time, enough people get together to build an open source alternative.
> And then like a snowball it eclipses everything proprietary that went before
> it._

I wish it did. The most obvious counterexamples that come to mind are
Microsoft Office and Adobe Photoshop. With maybe the recent exception of Krita
for the latter case, there are no known open-source alternatives that don't
suck _hard_ compared to the propertiary applications I mentioned.

I'm not really sure why that happens, but open source doesn't seem to scale
well when building large end-user applications.

~~~
EGreg
Maybe not end-user applications. But everything underneath them.

I also think that open-source needs some sort of end-user facing consulting,
the “last mile” at it were. A Wordpress expert or hosted service setting up
your blog. A browser like Chrome that uses WebKit or Blink underneath.

But the underlying free platform beats the proprietary ones in the end. Let
people build their little competing businesses on top of it if they want.

In the world I’m talking about (collaboration via contributions) everyone
would get UBI and be able to not contribute anything.

------
zeth___
Jupyter isn't a foil for mathematica. It has completely different use cases.

The straight one to one fight is between sagemath, or their rebranded cocal
site, and mathematica. In terms of ability to do things sagemath is the glue
between all the amazing open source math and science software that has been
written over the last 50 years.

[https://cocalc.com/](https://cocalc.com/)

But in terms of presentation by far the best is org-mode with sage, julia and
everything else tied in. One text file that can be emailed around, put under
version control, and can speak two dozen computer languages with babel, and
has latex support for both pdf and html output.

~~~
williamstein
Thanks for mentioning SageMath and CoCalc (I founded both of these projects)!
A minor clarification is that CoCalc is not a rebranding of SageMath, but is
instead a new web application whose goal is to make it very easy to
collaboratively use Sage, Jupyter, LaTeX, Julia, etc. In constract, Sage is a
more traditional open source software package, which people install on their
own computers. The goal of Sage is to be a viable open source alternative to
the core Mathematica computer algebra system (and also to Magma, etc.),
whereas the goal of CoCalc is to make all technical open source software very,
very easily accessible, mainly to students.

~~~
zeth___
Have you had a look at org-mode babel?

It's extremely interesting how they have managed to create an all plain text
environment that is in some ways[0] better than the current notebooks. Of
course it's not as user friendly and needs quite a bit of polish, and a major
refactor to separate the above use case from the tangle use case.

The tangle part of org is even more interesting as it's probably the first
fully literate system that allows you to completely document everything about
a software project. And since it's language agnostic I do mean everything from
install, compile, connect to the external databases, export the documentation
in every format under the sun and whatever else you can think of (including
docx through pandoc).

I'm going through the source code slowly to get it streamlined, but the code
is suffering from the typical lisp syndrome of "Everything is implemented
everywhere."

[0] Being able to run multiple languages on the same org file, controlling
environments for scripting languages, compiling on page for non-scripting
languages.

~~~
williamstein
Yes, org-mode is very inspiring. But, as you say, it's not quite user friendly
enough for some target audiences (e.g., beginning students).

