
Pyston 0.3: Self-hosting Sufficiency - kmod
http://blog.pyston.org/2015/02/24/pyston-0-3-self-hosting-sufficiency/
======
eiopa
What this actually does right now is run some internal Dropbox scripts that
exercises an unknown subset of Python + python stdlib, plus the perf
benchmark.

Unless I am missing something, the stuff you guys achieved have absolutely
nothing to do with "self-hosting".

I am curious - how does this compare to pypy, performance wise? Is there a big
win with this approach over pypy?

~~~
ctoshok
The use of "self-hosting" was addressed in post. I think its use is actually
right on the mark, but will grant you that it's slightly different than its
use in the context of a static compiler written in the target language.

It basically means "able to host its own development". This means something
different in different contexts, e.g. JS JITs "self-host" when some part of
their builtin code is written in JS. We share that property with JS JITs, but
in this particular case it means we can host our own build/test
infrastructure. So we're using pyston to develop pyston. ergo, "self-hosting
(qualified)."

\---

We're definitely slower than pypy at present - we're only a small fraction
faster than cpython after all on our benchmarks, and pypy is scary fast :)

I think we're still in the part of our work where the answer to the "big win
with this approach" question is "we don't know." We're confident that we
can/will be _much_ faster than we are now, but we have different constraints
than pypy, so it will always be something of an apples/oranges comparison.

~~~
eiopa
I see. I think it's a huge stretch :) That's like Go claiming they are self-
hosting because some of their tools are written in Go.

On a related note, I have to say that I still don't understand the point of
this project. Were you guys unhappy with pypy's performance or development?

I understand that in theory, the approach you guys are taking is incompatible
with pypy's, but I can't help but wonder how thing would play out if you
Dropbox put their weight behind PyPy. Maybe it could've been the start of
every Python programmer's wet dream: Python 2.8.

~~~
rian
You really could have just said "dream."

~~~
eiopa
I think "wet dream" captures it better.

------
rdtsc
Great work!

I remember when Unladen Swallow project was active, they were encountering
some issues with LLVM which slowed their progress down. They often had to stop
to fix LLVM.

Wonder if Pyston team got a chance to look or learn from Unladen Swallow's
idea or even use any of the code?

~~~
boulos
A lot of the barriers that Unladen Swallow / Reid and friends ran into were
fixed as part of their efforts (e.g. MB limits on emitted code, no gdb
support, etc.). Building a dynamic language runtime on top of LLVM is much
much easier today than it was when Unladen Swallow tried, and I'd say it's
partly thanks to their initial effort.

~~~
acdha
I'd bet they've also benefited considerably from the work done with the WebKit
FTLJIT project, too:

[https://trac.webkit.org/wiki/FTLJIT](https://trac.webkit.org/wiki/FTLJIT)

~~~
boulos
Right, I should have mentioned that. Though as the WebKit-related patchpoint
and other bits happened more recently, I had (still?) assumed that Pyston
hasn't gotten to leverage those much yet. Turns out I'm wrong:
[https://github.com/dropbox/pyston/blob/master/src/codegen/pa...](https://github.com/dropbox/pyston/blob/master/src/codegen/patchpoints.cpp)

------
shadowmint
Unless I've deeply misunderstood the code on github, this appears to be a
_python 2_ compatible implementation.

I have mixed feelings about that.

~~~
coldtea
Python 2 isn't going away any time soon. Less than 10% of the community uses 3
according to pip version usages...

~~~
shadowmint
I know, but the project page says:

    
    
        Currently, Pyston targets Python 2.7, only runs on x86_64 platforms, and only has been 
        tested on Ubuntu. Support for more platforms -- along with Python 3 compatibility -- is 
        planned for the future, but this is the initial target due to prioritization constraints.
    

...but if pypy has taught us anything, surely it's that implementing python3
after you have a working python2 implementation is _a seriously huge piece of
work_.

 _Especially_ if you're wholesale copying unicode naive cpython code into your
code base.

I'm deeply skeptical python 3 support will ever land for this.

Basically, this is 'modernize python 2 and make it faster'; there's been a lot
of talk about no one being willing to pickup and maintain a 2.8 version, but
this _is it_ , effectively.

Major backer, major new features.

So, lots of good things here, but there no doubt that its going to be divisive
in the community, and I'm not sure I really support that. ...promising py3
support nebulously at some point in the future doesn't fix anything.

tldr; If you ever plan on supporting python3, do it already. Otherwise don't
make fake promises.

~~~
BuckRogers
It seems the main feature to Python3 over 2.x is that it enables you to beg
projects to port to Python3. But Python3 people- _it 's opensource._ Isn't
that wonderful?

Python3 patches welcome.

~~~
shadowmint
Don't be ridiculous.

There's a reason the python 2.x line is 'patch only' by the developers.

Python 3 made fundamental changes to underlying string operations internally
for UFT16, which you can easily argue, was a huge mistake, but there you go.

You can't just 'patch python3 support' in. You literally have to rip out
anything that uses char * in the code base and replace it with a unicode
supported alternative, which is both more complex, and _breaks python 2
backwards compatibility_.

Pypy is very clever in how they handle this through rpython, which is why they
can kind of support both; but randomly dropping cpython 2.x code into the
project is completely not forward looking.

I'd be happy with: "We never intend to support python 3, sorry".

If that's the path you want to walk for all the complicated reasons you choose
it, fair enough.

~~~
BuckRogers
If you want Pyston to support Python3, get ta portin'.

I'm sure Dropbox would appreciate the help more than the soapboxing.

------
kasabali
Does anybody know if they considered a similar thing to what HotPy (2) [0]
did? I'm not anywhere near to understanding the implementation details, but
HotPy stood up most when I looked into different Python implementations.
Basically, it was a fork of an early development version of CPython 3.3 code,
with more bytecodes added (which were easier to optimize than the originals),
a tracing engine and some other things I don't understand. The best thing is
you get language compatibilitity for free and it is also binary compatible
with CPython extensions.

Unfortunately its development looks to be ceased.

[0]
[https://sites.google.com/site/makingcpythonfast/](https://sites.google.com/site/makingcpythonfast/)

~~~
IanOzsvald
HotPy is Mark Shannon's research project, he's not making a 'new better Python
for the masses', he's using it to test his assumptions about ways CPython
could be improved. Ask him about it if you bump into him at a python
conference!

~~~
kasabali
Thanks, it's more clear now. I misunderstood Pyston's motives and thought it
was just making Python faster rather than writing it from stratch.

------
montecarl
Is it expected that Pyston play well with numpy/scipy? This is the biggest
barrier that I see towards adoption of PyPy for numerical computing. If Pyston
can work with numpy and scipy then it would be a huge accomplishment.

~~~
ngoldbaum
Their focus is on getting the dropbox application working under pyston. It
would probably take contributions from someone outside of dropbox to make this
a reality. Presumably it will be easier on pyston than with pypy since pyston
has designed in better compatibility with the CPython C API.

------
rogerbinns
Does the C extension API remain the same? ie can I take an existing extension
and just recompile it, or does the code have to support a different API.

I ported my main extension to PyPy but had to leave bits of functionality out
because of missing parts of the API that CPython had that they didn't.

------
cpr
Is Guido still at Dropbox? Contributing to this effort?

If he is, that would certainly make the effort carry a lot more weight.

~~~
techdragon
Why?

Not downplaying how important our esteemed BDFL Guido is to Python, but I'll
be flat out honest, aiming their sights on Python 2 just further entrenches
its eventual place as the COBOL of 2050.

Python 2 is DEAD to me. It should be dead to you. It's Dead with a capital D.
Node.js wound up with all this fork nonsense in HALF the time Python 2.7 has
been stagnating the Python ecosystem.

Python 2 only code is nuclear waste grade technical debt.

I haven't written a line of Python 2 in 6 months after 2 years of slow
decline.

I now use Python 2 vs 3 as an interview question.

I run my tests on 3.4.x, 3.5-dev, and versions of HEAD that pass the Python
test suite.

Why does Guido working on Pyston do anything but hurt the future of Python by
legitimising the position that it's ok as a community of Python programmers,
to keep accruing this technical debt ?

~~~
svisser
Python 3 has no benefits if your Python 2 code works and many dependencies are
not Python 3 compatible yet. Why would a company even port to Python 3?

~~~
jsmeaton
There aren't that many dependencies without py3 support anymore. Of course, if
your project uses any then being stuck on py2 is a given.

I still don't think projects should port to python 3 unless they need to be
moved forward or continually developed. Greenfield projects should most
definitely start on py3 though.

------
phkahler
Does Pyston work with any of the GUI toolkits? If not, is it expected to at
some point?

------
andrewchambers
Thanks kmod, keep up the good work.

------
codexon
How is this different from pypy?

~~~
chrisseaton
One difference that was originally highlighted by the Pyston team as being
significant is that this is a method based JIT, where PyPy is a tracing JIT. I
think it also prioritises support for C extensions, which PyPy does not.

------
Scarbutt
But what if condoleza sneaks a trojan somewhere in the pyston code?

