
PyPy 5.1 released - sandGorgon
https://morepypy.blogspot.com/2016/04/pypy-51-released.html
======
BuckRogers
Wow. Just wow. Congrats to the PyPy team. It never stops.

I stopped calling myself a Python programmer sometime ago and started saying
I'm a PyPy programmer because this is the future of Python. The underlying
basis for PyPy has big implications for a number of dynamic languages though
so we should all be cheering it on and contributing.

For those who want better Python3 support, there's only ONE answer so no need
to keep begging. Donate more money to the project or donate your talents.
Patches welcome!

~~~
kzrdude
How can it be the future if Py 3 support is the side project?

~~~
sp332
After they get momentum by attracting people with their speed improvements, it
will be a lot easier to attract talent and money to tackle Python 3. The
reason Python 3 is on the back burner is that there simply isn't a large
enough audience right now. That could change if they keep increasing their
reach.

~~~
kzrdude
After they get momentum? But pypy has existed for a long time. They are in a
maturity phase now, surely.

~~~
LeifCarrotson
You don't get many cases of 20% memory consumption improvements in mature
projects...

~~~
wyldfire
"20% consumption" is more a factor of how much memory you have installed and
how much you intend to use than PyPy's maturity. I frequently use
malloc()/mmap() to consume much more than 20% of system memory.

EDIT: strike this, I misunderstood.

~~~
groovy2shoes
I think they're talking about a 20% _improvement_ in memory consumption
release-over-release, not complaining that it's consuming 20% of their memory.

------
sandGorgon
Why does PyPy not get support from the big Python based giants - the Dropboxes
or the Googles of the world. It seems to me that PyPy is struggling for
funding. PyPy has the potential to become the canonical Python - if the entire
team had their finances taken care of (and not have to do things like
[http://baroquesoftware.com/](http://baroquesoftware.com/) ).

Is there a particular reason for this ?

~~~
bkeroack
PyPy is in the unfortunate position of being both uninteresting to many of
those who rely on Python (numpy, etc and Spark work fine and are fast without
PyPy) and irrelevant to application developers who switched off Python because
it was too slow (to things like Go, Rust, etc).

(This isn't an insult to PyPy, btw--it's an amazing project that has
accomplished a lot. I'm just explaining why it doesn't get the support one
might expect.)

~~~
sandGorgon
but assuming the vast majority of python users are using it for web
application and so is Dropbox/Google, it seems to me that PyPy has a
marketing+PR problem than anything else.

~~~
pwang
That's not a fair assumption: a huge number of Python users are using it for
some level of data processing and data analysis. And that percentage is
growing each day, because Python is a very hot technology in data science.

------
canjobear
Are there major technical issues that make it hard to do PyPy for Python 3.5,
or is it just a matter of wanting to focus 100% on the fixed target of 2.7?

~~~
PeCaN
The main issue is that PyPy's commercial target audience (people who need fast
Python, particularly for numerical calculations) doesn't care about Python 3.
There's not nearly as much potential funding for Python 3 as for Python 2.

Technically, Python 3 is not any harder to implement than Python 2, it's just
useless.

~~~
mangeletti
I think you're partially right about the reasons (the other half of the
equation simply being the momentum of Python 2 efforts), but I think your last
clause ("it's just useless") is incorrect.

I can tell you for 100% certain that, if a project of mine could run on a
version of "Python" that was even twice as fast, I would switch immediately,
because 6 application servers could become 3. If the gains are even greater,
that's tremendous.

It's not just scientific computing that benefits, and in fact, a lot of the
scientific / numerical analysis done in Python is really done in C / C ++
anyway (think Numpy, et al which delegate to compiled code for much of their
expensive operations).

~~~
srean
I think it comes down to the question how much funding would such folks be
able to raise to support that effort. It is my understanding that PyPy is
quite resource starved.

------
rcarmo
Had to happen. New PyPy release, new 2vs3 feud in comments. Can't we all just
get along?

Anyway, my $0.02: I'm looking forward to test this with Hy (hylang.org) and
seeing if some of the module referencing is fixed. Makes for a great JITed
LISP already.

------
justantheruser
Are there plans to add Python 3.5 support?

~~~
wbond
Pypy3 is a separate project. From previous comments by the PyPy team, their
customers are almost all targeting Python 2, hence Python 3 doesn’t get nearly
as much attention.

Pypy3’s last release was 2.4.0 in 2014. It targets Python 3.2, which is
relatively unsupported compared to Python 3.3.

Here is a recent comment by a core member of the team on future Pypy3 work:
[https://news.ycombinator.com/item?id=11262150](https://news.ycombinator.com/item?id=11262150).

~~~
skolos
> their customers are almost all targeting Python 2

Odd statement. How can they have any customers who target Python 3 if they
don't support it?

~~~
Nkdo
The vast majority of python codebases are still in python 2, even though many
of the popular framework have been ported to 3, and there's no reason for most
of them to bother with the amount of work it can be to port to py3 when
they're happy with py2. Whether it be devs like Dropbox or open source
software like NVDA, they will stay in the world of python 2 for the years to
come. The amount of customers they could get by supporting python 3 is
definitely not worth the effort unless they get paid explicitly for it.

In terms of libraries, python 3 is "mostly" ready (though there are still
issues with highly popular ones like wxwidgets. Or amazing tools like Pyjamas,
which replicated GWT with Python instead of Java. Or Google AppEngine, which
still shows no sign of moving to 3.) but large codebases outside of highly
worked on web frameworks haven't moved on and there's not enough gain to
justify the effort it takes to port them. The Python 3 debacle caused a lot of
devs to also look at other languages for new projects. Old python 2 codebases
will stay python 2, but new projects might not necessarily be python at all.
The anger with Python 3 is real.

See also: [https://caremad.io/2015/04/a-year-of-pypi-
downloads/](https://caremad.io/2015/04/a-year-of-pypi-downloads/)
[https://www.reddit.com/r/Python/comments/45sm94/what_are_the...](https://www.reddit.com/r/Python/comments/45sm94/what_are_the_most_recent_python_3_vs_python_2/czzy5wa)

"Looking at these graphs, we can see that in the past year Python 3.x has
grown from roughly 2% of the total downloads from PyPI to roughly 5-6%"

I expect things like AppEngine to deprecate Python rather than port to 3,
when/if python 2 reaches EOL and not enough people gather to maintain or even
evolve that old branch.

Python 3 users are a vocal bunch, but the war they're fighting is a lost one.

~~~
andrepd
>The anger with Python 3 is real.

But why?

~~~
daveguy
Because here is a quick list of just over 100 things that have to be
considered to port and maintain compatibility between the two:

[http://python-future.org/compatible_idioms.html](http://python-
future.org/compatible_idioms.html)

Python3 is a different language than Python 2.7 and there is no compelling
reason to move from 2.7 to 3.

Most of what would have been compelling reasons are being handled in PyPy.
PyPy is the present and future of Python and that future is 2.7.

~~~
X-Istence
If you want to use just Python 3, after the initial port there is no issues
with having to maintain Python 2/3 compatibility.

Working in open source, there is going to be a time when I stop supporting
Python 2.x, and already I and others are building libraries that are Python
3.x first, are developed on Python 3.x and Python 2.x is an after thought in
testing.

PyPy will eventually have to catch up to 3.x, and no, the future is definitely
not 2.7.

~~~
Nkdo
No matter how hard you're wishing it to be true, does not make it so. PyPy
will "have" to catch up? in the name of what? PyPy is but one of the many
projects that are still thriving in the 2.x ecosystem, there's even a
competitor to PyPy that was launched 2 years ago that is also a 2.x exclusive:
Pyston. [https://blog.pyston.org](https://blog.pyston.org) Yes, a very new
implementation project that happens to focus on 2.7.

Python 2 gets all the better runtimes. Long after the end of life of the
official CPython 2.7, we will still be running 2.x programs, and doing it on
much better runtimes than CPython.

We've been hearing the same tirades of Python 3's future over and over again
by its partisans ever since its release in 2008. 8 years have passed since
then. 8 years will pass and Python 3 will still have nothing but a minority of
vocal open source users. Python will be treated as a legacy language the way
COBOL and BASIC are treated before py3 starts picking up any steam. That's
kinda what some companies are already doing, like Dropbox. Switching to Go
here and there, while keeping the maintenance of their very large 2.7 python
codebase with absolutely no intention whatsoever of moving that behemoth to
3.x. Why bother? they could write new software features, fix bugs and so on
instead of waste time on porting. Why waste so much time on porting when it
brings so little benefits? Programming is ultimately about problem solving.
Porting to python 3 doesn't make my software better. It's certainly not
running any more efficient either, that's PyPy and Pyston territory (Using
PyPy can allow you, under some circumstances, to seriously cut down the amount
of hardware you need). It doesn't magically add new features, or fix the bugs.

Python 3 will go down in history as the best example of what you should never
do while growing a language. People like to complain about C++'s complexity,
but unlike Python 3, C++ isn't threatened by its own past selves, and that's
despite adding features that are arguably far more compelling to look into
than the various differences between Py2 and 3. It's also very hard to regain
lost trust. The trust that was lost after what was done to Python with Python
3 can never be fully regained. I can write software in C++, Java, C#,
Javascript, Lisp, Perl 5 (which is still being worked on) and so on and have
an expectation not to have to go through something like Py3. To a lesser
extent, even Ruby, although it breaks compatibility from time to time, it
never went through anything as major in a one time fashion as Py3 did, which
is why the ruby ecosystem hasn't self destructed.

With moore's law being utterly dead, the fact that all the interesting
improvements to language implementation are happening only to 2.x rather than
3.x really contradicts your idea that the future lies in 3.x.

------
wyldfire
> Numpy: "Fix transpose(arg)...add broadcast..."

Does anyone know if there are some milestones in sight regarding packages
which use numpy? e.g. "working scipy or sklearn".

EDIT: BTW I forgot to add: tremendous thanks to PyPy team for your continued
work on this project. I benefit from it and always look forward to the next
release!

~~~
mattip
(PyPy dev here) Thanks. As our C-API support improves, we hope to be able to
support more of the scientific python stack with minimal upstream changes

~~~
sevensor
Hugely impressed that you now have broadcast and all ufuncs. As a user it
seems like these constitute a big chunk of what makes numpy what it is.

------
pdknsk
I wish PyPy supported multiprocessing. It does theoretically, but imposes a
significant penalty, so it's not in a practically usable state.

[https://bitbucket.org/pypy/pypy/issues/1538/](https://bitbucket.org/pypy/pypy/issues/1538/)

~~~
emidln
That's from almost 3 years ago. Have you ran the attached patch to check the
benchmarks with a more current release? I don't know if that's been fixed, but
it probably wouldn't take much time to test out.

~~~
pdknsk
I last tried my own code which uses multiprocessing a few weeks ago. If
anything, it gets slower with each release.

~~~
tachion
Is this the same case with threading (like multiprocessing.dummy)?

------
aidenn0
Is IBM involved in PyPy at all? Seeing PPC64 and s390x on a list of supported
targets is unusual for a language runtime.

~~~
gsnedders
They funded the PPC and s390x ports, as far as I'm aware.

------
haberman
Startup time is ok. "Hello, World" runs in something like 0.08s on my
computer. Not sure what this would be like for an app that actually has a lot
of code. Startup time is a metric I care a lot about, because it speaks to
whether it feels snappy to use for command-line applications.

But holy crap the binary is enormous. I'm getting 22MB of text, compared with
991kb for CPython.

------
abc_lisper
How compatible are third-party packages with PyPy? Is there a compatibility
matrix for all the packages on pypi?

~~~
wolf550e
[http://packages.pypy.org](http://packages.pypy.org)

The web server type stuff works. The scientific computing type stuff mostly
doesn't work.

------
davidchua
Pardon my ignorance but what exactly makes PyPy different from standard
CPython?

------
BuckRogers
Because the future is whatever you're building software in _today._

For the vast majority that's still Python2. For those starting new projects
that might be Go, Elixir, Python2 or Python3.

Just because some entity like the PSF declares Python3 the future doesn't make
it so. Even if some people buy into it and believe it. It has to take over
from the bottom up and its been floundering since 3.0 landed in 2008.

~~~
groovy2shoes
This attitude is the exact reason why Python 3 hasn't been getting the uptake
that was expected back when it was released. People who sit around and say
"the future is today, and I already have Python 2 today" go on to start even
totally new projects in Python 2, when they very well could and probably
should be using Python 3. As the sibling comment points out, it's just
laziness (and rationalization). There's no two ways around it.

Like it or not, the fact of the matter is: for Python programmers, Python 3 is
the future. It's really only a matter of time, even if it's much more time
than was expected.

~~~
BuckRogers
You're confused. The unfortunate matter is that it has few to no technical
merits. It defined technological churn with total absence of technical
innovation. THAT is why it failed. Not because "everyone is lazy".

Python3 was lazy. Demanding that everyone including PyPy port, is lazy. Let
Python3 live or die based on its technical merits like every other piece of
technology.

Not to mention I wish the sword you're wielding cut both ways. You aren't
entitled to demand someone else's labor any more than your buddies that litter
every PyPy post demanding better Python3 support are. I hope you're harassing
them for being lazy.

You made your own choice. You should have picked based on technical
superiority rather than propaganda. To be honest with you, you should be
ignored as you chose VERY poorly. Don't dare tell someone else what to use.

There are so many better choices to build new software in today than Python3.
Python2 on PyPy is one of those. Those people can still port to Python3 if it
ever does make sense. Even if that idea clearly threatens some of you, that's
the smart play.

~~~
groovy2shoes
Yes, I have made my choice. And frankly, it isn't Python at all.

But the fact of the matter is that Python 3 is being actively developed,
whereas Python 2 is in maintenance mode, and for how much longer? One of these
days, Python 2 will become unsupported, and at that time you either finally
move to Python 3, or you fork -- at which point you're no longer, strictly-
speaking, programming in Python.

You're right that it's also lazy for people to simply demand Python 3 support
from the PyPy project without being willing to invest in the effort. I have
nothing to add to that, really.

I should also add that I'm probably one of the laziest programmers to ever
exist, so I'm not trying to demonstrate superiority or anything like that. I'm
simply pointing out that refusing to use Python 3 just because it hasn't seen
much uptake only perpetuates the problem.

~~~
BuckRogers
You'd be correct. The distinction is whose fault is that, the lazy Python3
core dev team or the lazy userbase? We'll just assume everyone is lazy in
effort towards equality. Python3 was created without any backwards
compatibility to make the core dev teams job easier, nothing more nothing
less. LAZY?! Yes. That's their prerogative, but the rest of us don't have to
use it either.

When I say lazy in this post, I mean simply acting on one's self-interests.
Because that's what every party involved here is doing.

It's important to note the difference between Python the language, and
CPython. Using Python2 doesn't mean you're using some deprecated language,
there's virtually no difference. That's what made the break such a shame
honestly. Moving to Python3 with new code is easy. Most people just have few
great reasons to now with their existing projects.

People need to stop harassing the community and start harassing Guido van
Rossum and the core dev team. Usually when people make a mistake, people fix
it. GvR and his rogue band of developers have not corrected the Python3
mistake in any way. They've simply continued their mistake for so long, that
people are becoming twisted thinking that it's EVERYONE ELSE who is at fault!
Amazing really.

CPython2 maintenance mode is just a way for the core dev team to maintain
their grip around this whole debacle. If they had the guts to dump it in 2015
they would've lost complete control of the situation. It wasn't out of
compassion as they say. Because they simply removed the incentive for a 3rd
party to swoop in and takeover CPython2.

Even that is backfiring now. People[0] are noticing how Python3 is turning
into feature soup, which is never an improvement. Thus Python2's conservatism
is starting to be seen as an asset.

As far as CPython3 being under active development, so is PyPy and it's
superior to CPython. The language doesn't change too much at this point.
Python is Python for the most part. When the time comes for someone on 2 to
move to 3, it will be a non-issue. They're not "losing out" today in any way
by acting in their best interests using 2.

Python3 has nothing that didn't already exist as a 3rd party library, nor does
it have anything that doesn't exist in form of a backport. Thus technical
innovation contained within is zero.

Programmers aren't stupid. And I think the PSF folks think we are. Or just
blatantly disregard the community that truly made Python what it is.

The existing Python2 community is overtly disrespected when called "lazy" and
all this type of harassment that goes on. I know I'm in the majority opinion
because upvotes don't lie.

I'm just a guy who likes "Python", the language. I can't blame you for not
using it and honestly, what a mess Guido and his core dev team has created
right?

Python is my preference but I'm not held hostage by the PSF or Python3. I use
Python2, Python3, Boo, PyPy. I don't hate Python3 and don't know anyone who
does. It's just no slam dunk, and that's the core development team's fault-
and they may lose their battle against Python fans as a result. I do _highly_
dislike the groupthink about Python3, with a portion of the Python community
on this topic.

One more thing to add, is Python3 inevitable? No. It just fragmented the
Python ecosystem. Nothing more. We all won't be on a single version again. GvR
and his band of rogue developers should be ashamed of themselves. Again,
change happens from the bottom up. Not top down. They failed the community and
should abdicate their roles but I'm afraid their egos are too big.

[0][http://learning-python.com/books/python-
changes-2014-plus.ht...](http://learning-python.com/books/python-
changes-2014-plus.html)

~~~
collinmanderson
> The distinction is whose fault is that, the lazy Python3 core dev team or
> the lazy userbase?

I agree that Python core devs should have not made such breaking changes, And
hindsight is 20/20, even for them. I also am a bit annoyed that they've been
pretty stubborn about adding additional compatibility layers between the
versions. (If six (maintained by a core dev) can help with compatibility, why
can't the standard library have those shims too? :)

I also like Go's approach to unicode better than Python 3's approach.

"harassing" the core devs may have been helpful in back in 2008-2012, however,
at this point it's a little late to start complaining. I doubt it will help at
all.

> Python2's conservatism is starting to be seen as an asset

I totally agree. It's an unfortunate problem.

> is Python3 inevitable?

Yes. It's only a matter of time.

\- Django is planning on removing Python 2 support from master in January.

\- Nearly all popular libraries are compatible or have compatible forks.

\- Large python projects, like OpenStack are putting in the work to be Python
3 compatible. [http://blogs.rdoproject.org/7894/status-of-python-3-in-
opens...](http://blogs.rdoproject.org/7894/status-of-python-3-in-openstack-
mitaka)

\- Ubuntu and Fedora now ship without /usr/bin/python available by default.
RHEL/CentOS 8 may very well do the same.

\- Debian has plans of removing Python 2 completely from it's repository not
long after PSF drops support.

~~~
BuckRogers
I agree with a lot of what you're saying. On Django though (or any library
that intends to drop 2.x support), I feel if they truly go through with it,
they'll be in for a shock.

It will more than likely hurt them more than it hurts Python2 (I also don't
understand the obsession with people wanting to kill or hurt Python2). People
and companies may be more likely to start new projects in Flask or another
alternative, maybe even fork Django which would be a disaster for them.

But as of today, Django's roadmap has the LTS release that supports 2.x as an
unknown EOL. So it won't be anytime soon.

Overall I tend to agree that Python3 will eventually take over. That wasn't
what I was arguing and I never said otherwise in my posts. My problem is with
the harassment over Python2 folks to port and all the associated nonsense.
That's the part that is more counterproductive than anything.

"A matter of time" is a really bad last leg to stand on as well, because time
has an awfully broad range. :)

As far as the other points, I have doubts about the impact of Ubuntu or others
defaulting to Python3. That won't stop me from building for PyPy, I can say
that at least.

In general I feel that I'm on a similar page to your stance though. I do use
Python3, and have since the 3.0 release in testing alone. For someone who has
tried every release and found performance issues and bugs galore, it's a
testament that I (and others) haven't given up completely. I think many have.

