

Unladen Swallow is dead - ot
http://groups.google.com/group/unladen-swallow/msg/bd4e6f935c23252a

======
jnoller
No, it's not dead, and I'm sad that this is sitting at the front page, because
it's so misleading.

The fact is, it's stalled
([https://www.google.com/buzz/bcannon/bZDN1jNZ3uC/Is-this-
fina...](https://www.google.com/buzz/bcannon/bZDN1jNZ3uC/Is-this-finally-
going-to-push-CPython-into-the)) - <http://www.python.org/dev/peps/pep-3146/>
was accepted. This means that a branch was made with the intent of merging US
into Py3k. Unfortunately, this proved to be a greater task then originally
planned. Stalled != Dead.

To quote Collin: " _Unladen Swallow is stalled because merging the code into
the py3k-jit branch plain sucks: lots of new compatibility work to be done,
lots of corners we had to un-cut, lots of drudge and suffering. It doesn't
help that Google is full of far more interesting and influential projects._ "

So add in the fact that the resources Google had "assigned" to the project got
moved/reassigned onto something else, and what you have is a very stalled
project. Collin has stated that they hope(d) that other python-core devs
would/will step up to help, but everyone is pretty tapped out as it is.

I'm as disappointed as anyone that it's stalled - given it's compatibility
with C extensions, potential speedups and future potential, it has been one of
the projects I've had the most hope for.

That said, PyPy is also coming along; when they hit 2.7 compatibility and they
have drop-in replacement capability, the ecosystem will be _very_ interesting.
Competition in the interpreter space is _good_ , and I feel that Unladen's
aggressiveness helped spur/move PyPy along.

That said Unladen (in my mind) remains the CPython interpreter's best way of
moving into the future. It still has the most promise for _that_ code base.

~~~
ot
jnoller, I may have indulged in catchiness at the expense of accuracy and I am
sorry if the title is misleading.

I hoped that this thread would raise some discussion about the status of US,
as, after a lot of publicity on project launch, there have not been many
official statements from the team. I have followed the project since the
beginning and saw the exponential decay of traffic in the mailing list and
commits as the promised performance improvements seemed less achievable (but
it has to be said that the US team contributed to LLVM a lot of useful code,
like gdb debugging of JITted code).

So my question is: why is US relevant since it hasn't achieved any significant
performance boost? Why should we believe in the "potential speedups and future
potential" if there is no evidence that supports them?

I myself had done some experiments with implementing a python JIT with LLVM
some time before US was launched and had the same results, 10-15% improvement
on microbenchmarks at expense of huge memory increase and compilation
times.(see [https://groups.google.com/d/topic/unladen-
swallow/bqf9TzWHht...](https://groups.google.com/d/topic/unladen-
swallow/bqf9TzWHhts/discussion) )

It seems that the bottleneck is not in the interpreter loop, but in the
CPython runtime API. I am afraid that, if the interpreter has to stick to
CPython source compatibility, it cannot get rid of the overhead. But I would
be very happy to be proven wrong.

~~~
jnoller
When asked - on the mailing list, on google buzz, or anywhere else, Collin and
Jeffery and others have been very open about its status. They've always dealt
us (as the community) a fair hand about the status of the project, and the
path moving forward.

Your indulgence in catchiness is annoying, as it will be echoed (and it
already has been) all over the place with breathless drama "Look! It's Dead!"
when, despite languishing, the code is still there, still functional, but
starved of resources. The editorializing is frustrating, misleading and
overall, I think harmful.

As for why it's still relevant: Easy, it's the only _currently visible_ way of
moving the CPython interpreter forward from an evolution standpoint. US does
show a series of speedups, one thats useful for CPython. To quote:

" _A JIT compiler is an extremely flexible tool, and we have by no means
exhausted its full potential. Unladen Swallow maintains a list of yet-to-be-
implemented performance optimizations [49] that the team has not yet had time
to fully implement. Examples:_ " - <http://www.python.org/dev/peps/pep-3146/>

I would recommend: if you want to move anything forward, or help, rather than
make posts like this, jump in on the
<http://svn.python.org/view/python/branches/py3k-jit/>

Yes; part of the reason the current interpreter is limited _is_ due to
backwards compatibility, but I'd rather have a limited JIT with some
performance gains then none at all.

The fundamental issue is that Jeffery, Collin and Reid - the founding three
are "off in the woods" - and they're the only ones who understand the code
enough to move it forward right now. PyPy's success(es) are orthogonal to the
evolution of the CPython interpreter code base.

~~~
ot
I am sorry this thread is turning into a flame... I would like to make clear
that I am not questioning whether or not US is good thing to happen to
CPython, just that it is now a project without maintainers (the post suggests
that they will stop working on it after the py3k-jit merge) and without an
active community. As you said it is a very complex piece of code and even if
it is "still functional" it is definitely not ready, as the performance gains
are not substantial.

I have nothing against calling it "stalled", but IMHO it is just an euphemism
for "dead".

Also I don't understand why people at Google decided to stop investing on it,
the cost of 3 full time engineers is nothing compared to the millions they
would save with a more efficient python implementation, given their huge
python infrastructures (App Engine, for example).

~~~
jnoller
If this is a flame war, then it's a pretty tame one :)

As for Google's intentions, I've learned (as it has been explained to me time
and time again) - never attribute smaller decisions to a larger strategy or
"plan" of Google's. It probably not that they decided to stop investing in it,
more than "something more interesting came up".

I do wish that we could have funded work on core/the interpreter. That would
be nice.

------
jacquesm
This sort of thing is one of the reasons why I'm very careful in picking the
pieces that make up the tools that I earn my living with.

Left right and center you're bombarded with 'use this', 'use that', but most
of these projects end up not having much staying power and if you depend on
them for the running of your business then sooner or later you might end up
having to support them.

When I choose an open source component these are the things I like to see in
the eco system around it before deciding to jump in (obviously not every
project selected will have all of these, but more helps):

\- multiple maintainers, active response to patches being sent in

\- broad support, using open standards

\- preferably a drop in replacement available

\- lively community of people willing to help each other out

\- two years track record with a good roadmap for future support

\- people working on it because they like it, not because they get paid to
(this one is probably very counter intuitive when you're looking at this
problem from a business point of view, but it is my opinion that people
working on something because they like it are automatically in for the long
haul).

Every time I've been seduced by new, hot and sexy technology I've come to
regret it sooner or later. Old, boring and solid seems to win the race every
time.

Running a small shop means that the investment you make in a tool is one of
the most important decisions you can make, the wrong decision could cost you a
lot of time and/or cripple your company.

I hope that everybody using unladen swallow is able to easily transition to a
replacement.

~~~
cdavid
There is no need to transition from unladen swallow since almost nobody was
using it. It was a fork of cpython, and the speed improvements were so
negligeable that I doubt anyone sane would try to use this without mastering
the - heavy - technology behind.

I was actually quite shocked when unladen swallow was somehow almost accepted
for python 3.2 at some point, given how bad the performance results were: if
you add C++ + llvm and all the new infrastructure/complexity it brings, you
rather have good numbers, not something like 10 % speed improvements which was
the order of magnitude last time I checked.

It did bring useful code to python, though: in terms of benchmark code, but
also proving that speeding up python while keeping cpython compatibility (vs
just python compat) is actually pretty hard (which was expected, but they
confirm the suspicion).

~~~
rbanffy
> I was actually quite shocked when unladen swallow was somehow almost
> accepted for python 3.2 at some point, given how bad the performance results
> were

Memory is getting cheaper. I was looking forward the GIL removal (at first),
but I'll be happy with any multi-threading improvement.

OTOH, when I really needed it, the multiprocessing module did the job nicely.
And it also prevents some very hard to catch bugs.

~~~
cdavid
Unladen has never intended to remove the GIL - their primary goal was to keep
compatibility with C extensions without even recompiling IIRC (google seems to
rely a lot on python extension around their infrastructure), and removing the
GIL will break many, many extensions: the only known way to remove the GIL
without cause a huge slow down is to remove reference counting, which C
extensions rely on, and also most C extensions rely on the fact they don't
need to be reentrant.

~~~
gthank
Removing the GIL was one of the explicit goals as originally conceived, but
was dropped when it was realized how difficult it would be (source:
[http://code.google.com/p/unladen-
swallow/wiki/ProjectPlan#Gl...](http://code.google.com/p/unladen-
swallow/wiki/ProjectPlan#Global_Interpreter_Lock)).

------
cawhitworth
Reading the thread, it seems there's still an intention to roll the JIT into
py3k at some point, but the project devs don't have the time and the community
doesn't seem to have the inclination to take over the work. I think calling it
dead might be overstating things slightly.

Or, seeing as this is Python we're talking about: It's not dead, it's just
resting.

~~~
ot
The thing is, in two years the project was able to deliver only a very small,
say 1.2x, speedup, with a noticeable increase of memory (sometimes 3x).
Against Python 2.6. I wouldn't be surprised if Python 2.7 is actually faster
on some tests. Is it really worth keeping working on it and merging?

~~~
jnoller
Part of the reason Python 2.7 is _faster_ on some benchmarks is because of the
patches the unladen swallow team pushed upstream into core.

------
rfugger
With PyPy moving ahead so well, I wonder if there's much point to Unladen
Swallow anymore. Weren't they always expecting to be a stopgap until PyPy was
ready?

~~~
rbanffy
Last time I checked, PyPy wasn't compatible with modules written in C. That's
a show stopper for many of us.

~~~
dchest
_Can I use CPython extension modules?

Yes, but the feature is in alpha state and is available only on trunk (not in
the 1.2 release). However, we'll only ever support well-behaving CPython
extensions. Please consult PyPy developers on IRC or mailing list for
explanations if your favorite module works and how you can help to make it
happen in case it does not. We fully support ctypes-based extensions,
however._

[http://codespeak.net/pypy/dist/pypy/doc/faq.html#can-i-
use-c...](http://codespeak.net/pypy/dist/pypy/doc/faq.html#can-i-use-cpython-
extension-modules)

~~~
rbanffy
Wow! That's really cool!

~~~
stuaxo
Their working on it...

------
acme
Unladen Swallow looked very promising, came out with lots of excitement and
news. It overannounced and underdelivered. I tend to prefer underannouncing
and overdelivering...

~~~
rbanffy
If you never fail, you need harder things to do.

~~~
JoachimSchipper
On the other hand, if you never _publicly_ fail you may just be prudent. Or
paranoid. Or depriving yourself of some beneficial pressure. But it's not
quite as clear-cut as you make it seem.

------
pin4444
Nice editorializing there. I suggest readers who care should read the entire
thread, and be careful when they see future posts from "ot".

~~~
csytan
I read the entire thread and it seems to me that the project is dead (or not
very far from it).

    
    
      Jeffrey and I have been pulled on to other projects of
      higher importance to Google. Unfortunately, no-one from
      the Python open-source community has been interested in
      picking up the merger work, and since none of the original
      team is still full-time on the project, it's moving very
      slowly. Finishing up the merger into the py3k-jit branch is
      a high priority for me this quarter, but what happens then
      is an open question."
    

When working on open source, especially something as complicated as this, you
need at least one dedicated maintainer. Looks to me that they are trying to do
the responsible thing and tie up any loose ends.

------
eliben
Are we reading the same page? I don't see anything saying it's dead. Perhaps
out of focus, but there's still a plan to merge.

~~~
briancurtin
I don't either. This same thread gets posted on Reddit's /r/Python seemingly
once every few weeks and everyone freaks out all over again.

------
zacharyvoase
It’s not dead, it’s just pining for the fjords.

------
pingswept
Is there a wise one among us who could summarize the characteristics and
status of the various Python interpreters us Python amateurs should be excited
about?

~~~
ronnix
Here is a comprehensive list of Python implementations:
<http://wiki.python.org/moin/PythonImplementations>

------
scrame
Funny, I was just talking about them not being able to remove the GIL today
while working through a tornado app.

I wonder what the future holds for pypy.

------
ronnix
It's not dead, it's sleeping.

