
The Making of SciPy 1.0 - apples2apples
https://www.numfocus.org/blog/scipy-1-0-16-years-in-the-making/
======
BucketSort
I love SciPy and the surrounding pythonic scientific computing tools!
Especially Jupyter notebooks! I've been playing around with SymPy a lot lately
and love that it pretty prints latex in notebooks. Here's a nb I did recently
on the taylor series demoing quite a few features:

[https://github.com/DanburyAI/SG_DLB_2017/blob/master/Noteboo...](https://github.com/DanburyAI/SG_DLB_2017/blob/master/Notebooks/MemberNBS/NumericalComputation/The%20Taylor%20Series.ipynb)

------
knolan
This a great resource for engineers to wean themselves off Matlab. Maintaining
Mathworks licenses is expensive and they can be assholes about license usage.
We recently purchased some additional toolboxes and all they could do was
bitch about how we had deployed our existing licenses. If you don’t like it
why do you allow it?

In addition, recent versions of Matlab have broken some common plotting tools,
contourf in particular. Where this once produced elegant and light weight
results it now generates a tessellated mess with tiny gaps between elements
that look awful in paper manuscripts and baloon the file size for no good
reason. The Mathworks response? It fixes some edge case.[0]

I find many of students we take on as interns often lack more than rudimentary
coding skills making the jump to Python easier.

[0]
[https://uk.mathworks.com/matlabcentral/answers/162257-proble...](https://uk.mathworks.com/matlabcentral/answers/162257-problem-
with-patch-graphics-in-2014b-splits-in-two-along-diagonal)

~~~
ChrisRackauckas
I don't see a reason for MATLAB anymore now that there's Julia. It takes the
good parts of the linear algebra syntax and throws out the rest.

~~~
knolan
The only reason, if you can call it that, is academic inertia. It’s is taught
as part of most engineering degrees and is accessible enough for students to
stick with it.

------
allenz
Summary: SciPy added Windows binaries for pip and formalised their
organizational structure.

------
numfocusfnd
SciPy 1.0 — 16 Years in the Making
[https://www.numfocus.org/blog/scipy-1-0-16-years-in-the-
maki...](https://www.numfocus.org/blog/scipy-1-0-16-years-in-the-making/)

------
codewardenh
Working with these libraries and the ecosystem has been great. We tried Julia
for a while but found it sorely lacking whereas python has such a great
community and history and wealth of experience and wisdom to share.

~~~
ChrisRackauckas
I have had the opposite experience since most decent Python packages seem to
have their own implementation of a parser, JIT, etc. in order to have any
speed, making them monolithic monsters that are hard to contribute to and are
hard to modify. Python's ecosystem seems to handle the basic cases very well,
but when I wanted to "go to research land" in pretty much any scientific
computing subject (mathematical optimization, numerical linear algebra, or
differential equations), I quickly hit a wall that would require I write one
of these monoliths myself in C++. I haven't hit any walls in Julia, but then
again if you stay in the basic standard equations/models YMMV.

But SciPy is a great project and everyone can learn something from their
successes.

~~~
m_mueller
I don't really get your point about monoliths in this specific usecase. Let's
say you're exploring with NumPy. What stops you from

(a) writing your research as a bunch of Fortran or C kernels and intrgrate
them with automatic bindings such as f2py? Especially Fortran is a great fit
for NumPy datastructures - because they are the same.

(b) Use high performance python environments like Numba, NumbaPro (GPU) or
even Cython?

~~~
ChrisRackauckas
>(a) writing your research as a bunch of Fortran or C kernels and intrgrate
them with automatic bindings such as f2py? Especially Fortran is a great fit
for NumPy datastructures - because they are the same.

If I have to write everything important and difficult in Fortran or C kernels,
then I'm losing the productivity of the higher level language. And in this
case it would be writing kernels from scratch instead of tweaking high level
package code if I want to change data structures and implement new algorithms
for existing problems. This makes it hard to directly re-purpose the internal
linesearch code from the SciPy optimizers without Python in-between into my
own Fortran optimizer, or add new projections and stuff to existing SciPy
implementations and have that be a separately maintained package but dependent
on SciPy.

Even just talking about this starts to sound like a mess from the past, but
fortunately this problem is already solved by Julia so there's no reason to
have to deal with it.

>(b) Use high performance python environments like Numba, NumbaPro (GPU) or
even Cython?

Well the biggest thing against Numba is once again you have to write
everything from scratch if you want to do things non-trivial. They finally
have a way for you to extend Numba to add a class, but you don't by default
get the whole Python standard library compatible with it, and classes defined
in user's packages aren't automatically compatible with it. So you have to
start re-writing basic data structures and other types to get the full
algorithm JIT'd. And Python's standard library isn't all setup with
@generated_jit to then dispatch everything effectively. So you end up having
to build tons of things on your own because of the lack of library support,
and it adds development time. Then there are restrictions on allocations, no
easy way to do distributed parallelism, take in functions to my functions to
inline them (this is important for writing things like ODE solvers, but with
Numba you have to do a hack to make this almost work but it won't optimize
this well), etc. Essentially it hits a feature wall when it gets past
microbenchmarks and to real programming. This all hearkens back to your first
comment:

>Let's say you're exploring with NumPy

Exactly: if you're content with building everything off of arrays yourself
then Numba will do fine. But its limitations significantly impact the ability
to compose full packages in native Python with constant propagation,
interprocedural optimization, etc.: all the stuff you you depend on static
languages doing in order to optimize large-scale software. On top of that,
it's a difficult dependency to install. So in the end, it's great for JITing
little blurbs, but I quickly hit the edge when trying to fight it.

Cython is similar to Numba, where you can setup your own fused types and
extension classes, re-writing the standard library, etc., but now you've taken
to writing essentially the stdlib for a language (or at least the parts you
need) just to be allowed to use this "easy accelerator" for a demanding
project.

If it was 10 years ago these issues would be okay, but these days it stands
out as a productivity and performance barrier. Julia has already solved this
problem, and hell even LuaJIT is easier to build this kind of software with.
Or if you're going to fuss around with compiled code, we can use D or Rust, or
even go all the way back to C++ and Fortran. In fact, it's quite telling that
even fairly recent big "Python" projects like TensorFlow, PyTorch, FEniCS,
etc. all built their whole package in C++ and made Python bindings instead of
trying to use these accelerators. Meanwhile people are punching out large
Julia projects are using pure Julia without hesitation.

To an end user of packages it really doesn't matter how it was implemented,
but when your work and research is in developing these kinds of large software
projects, these limitations make a huge difference.

~~~
m_mueller
Thank you for the expansive answer - I can really see where you're from now
and I agree with your issues in Python. It's interesting to me that you go to
Julia as a solution (I mean that honestly, I want to play with it when an
exploration phase comes up).

------
kenaiphatdu
Great news. It's really good to have an alternative for the expensive Matlab.

