PyPy are doing such incredible work and they seem to ask for very little in the way of funding to make it happen (and they're very explicit about where they're going to spend the money).
Why is it that there aren't more donations from the industry? Is it just a marketing issue? Do they need to do a snazzy kickstarter video to build the hype?
There are plenty of companies that could benefit from PyPy doing well though. How many python servers do youtube / reddit etc run? With some investment in PyPy they could reduce their on-going costs. Youtube could easily dump $100k into the PyPy project, so what's stopping them? (I suspect the answer in that case is that Google just aren't interested)
I'm in a startup that doesn't have money to invest in PyPy, but if we we had more cash we'd definitely look at putting some into PyPy (though our use case is the hairy NumPy side of things).
I guess in the case of PyPy it's maybe less interesting to individuals but maybe there's some way of reframing it to make it interesting.
In a way, I'm sort of the target market here, because I use python all the time and I haven't donated to PyPy. So why haven't I donated (other than being quite broke because I'm starting a business)?
I think part of it is the urgency. There are 3 specific campaigns you're running  to get funding, along with funding for the general project itself. And the campaigns themselves are great - you have good overviews of the issues that you're solving, a realistic approach and a proven track record of being able to execute. But I don't know when those campaigns started and what the cut-off date is. They sort of feel ongoing and it feels like there's no loss in not donating right now. Compare with something like kickstarter, where you have a date by which if there hasn't been enough funding you may end up losing out on something really cool.
Sorry for the rather rambling brain-dump, but maybe there's some useful perspective here.
 http://pypy.org/tmdonate2.html http://pypy.org/py3donate.html http://pypy.org/numpydonate.html
One of those things can be to provide a very clear road-map tied to donations and then follow up on it. That's what so powerful with these crowd funding campaigns. People feel like they are making a difference though at times they are essentially buying something.
I don't think you necessarily have to "compromise your values" either. You're probably already limited by money, so add some planning and just be honest about it. "With this amount, this will happen" etc. As always of course "under-promise and over-deliver".
That's mainly for individuals though. Companies are somewhat the same, but they want things that fit into their budget.
Several linux distros have much of their customization features written in python.
Microsoft seems to be trying VERY hard to change their course and attract more developers back to their platform(s). They're clearly willing to make strategic investments in open source projects.
I've found that PyPy works wonders for the vast majority of python code I've thrown at it. In order to close remaining gaps, funding is key.
RPython/pypy is kind-of a competitor to the CLR (especially with DLR) so unless it's for a CLR/DLR backend I doubt Microsoft would show that much interest.
I may be silly but why don't you contact a professional lobbyist ? Those people know how to get money out of those big corps and could maybe be used for this good cause.
The marketing angle could be also interesting. The Big G would pay a little to appear benevolent to the community. Heck, the new Microsoft (which use Python alot) would pay to continue building their new image.
And ask the guy from LuaJIT, if he can do it...
Compare this with much less ambitious projects where developers gets funded when some company or other commits to it. Perhaps the Euro-centric development holds them back here from getting some of well earned Silicon Valley cash?
I wish I had the slighest idea how to help them, apart from pitching in some beer money. They do truly great work.
Something in the release notes caught my eye:
> The past months have seen pypy mature and grow, as rpython becomes the goto solution for writing fast dynamic language interpreters.
I asked this question on the Perl 6 thread from a few days ago but didn't get an answer. Does anyone know why on earth the Perl 6 folks created yet another dynamic language VM+JIT with MoarVM instead of taking advantage of all the great work done with PyPy? Does anyone know whether PyPy was even considered as a target before writing MoarVM?
NQP is a subset of Perl 6 which the Perl 6 compiler is written in. That compiler takes a program written in Perl 6 and converts it into NQP. That NQP program is then compiled to different targets such as JVM, Parrot etc. This is my understanding, of course.
> I think MOAR was designed specifically to run NQP efficiently.
> NQP is a subset of Perl 6 which the Perl 6 compiler is written in.
> That compiler takes a program written in Perl 6 and converts it into NQP.
No. moar is a vm backend for nqp, with a traditional bytecode interpreter, a jit, gc, ffi, and binary format.
>That NQP program is then compiled to different targets such as JVM, Parrot etc. This is my understanding, of course.
No. Of the current three perl6 backend moar, parrot and jvm is the fastest, but has problems with its traditional threading model. It does not scale linearily up the number of physical cores. It needs locks on all data structures, but this is still better than with perl5 or pypy. parrot does scale linearily, has locks only in the scheduler, but needs compiler support to create writer functions, only owner can write, access through automatically created proxies. The jvm threading model is also faster and well known, but has this huge startup overhead.
perl5 has to clone all active data on thread init and creates writer hooks to copy updates back to the owner. this is a huge startup overhead, similar to the jvm.
Overall, would you expect any c/perl compiler dev to switch over to rpython to back your perl6 vm? Writing a jit for a good vm was the matter of a summer, and writing the compiler optimizations needs to be done in the high-level language of choice to be more efficient and maintainable.
You should better look over to rperl to compare it to rpython. This is the similar approach. A fast restricted optimizable language subset, which compiles do typed C++, with fallbacks on unoptimizable types to the slow perl5 backend. No need for a jit with its compilation and memory overhead. A jit only makes sense for dynamic profile-guided optimizations, as in pypy or v8.
Even if the simple static jit's as in p2 or luajit without much compiler optimizations still run circles around those huge battleships in the jvm, clr, v8, pypy or moar. optimized data structures still are leading the game, not optimizing compilers.
>No. moar is a vm backend for nqp, with a traditional bytecode interpreter, a jit, gc, ffi, and binary format.
But surely NQP is a "language" and Moar is a vm ?
It's also not quite fair to start Perl 6 at announcement and PyPy at their first release (they were organized enough in 2004 to be applying for EU funds).
I imagine at some point a PyPy targeted Perl 6 will appear. It may or may not be based on the Rakudo implementation as a target for NQP (jnthn was able to get initial JVM support for NQP completed in a summer), or another implementation that targets PyPy through rperl directly. It would be really interesting to see what Perl 6's automatic opportunistic auto-threading of list operations and PyPy's STM work could do together.
The comment I replied to frames it such that PyPy wasn't an option for MoarVM because it didn't exist when things got started.
However RPython is not thread-safe and the GC it provides is not thread-aware, so you'd have to provide your own thread-aware GC (or a big GC lock)
But spawning threads in your Perl code comes with the same set of problems as anywhere else, it's easy to create races or deadlocks, and Perl happily lets you do these things if you ask it to.
1. Python is used to setup/glue together code that is written in a compiled langugage.
2. Someone writes a code by starting out writing everything in Python, then profiling and identifying the slow parts, and then writing these in a compiled language.
Edit: example of 1): HOOMD-blue, example of 2): PyClaw.
Consider "a + b + c + d". For large arrays, the problem is that it creates many temporary results of same size as original arrays that must be streamed over the memory bus. And since FLOPs are free and your computation is limited by memory bandwidth, you pay a large penalty for using NumPy (that gets worse as expression gets more complex).
Or "a + a.T"... here you can get lots of speedup using basic tiling techniques, to fully use cache lines rather than read a cache line only to get one number and discard it.
And so on. For numeric computation, there are large gains (like 2-10x) from properly using the memory bus, that NumPy can't take advantage of. So you have projects like numexpr and Theano that mainly focus on speeding up non-loop NumPy-using code.
If it's lots of small operations, I think pypy can inline them and you might see a significant speed up.
What would be the best way to profile this thing under PyPy to see where it's spending time? I'm barely familiar with Python profiling in general, and totally unfamiliar with PyPy specifically.
As for how I'm using it, I belong to a glider club. For every weekend day we operate (which is every weekend from around the end of February through mid-December) we need to assign four club members to carry out various tasks for the day. (Specifically, we need a designated tow pilot, an instructor, a duty officer, and an assistant.) I'm the scheduling guy, and I wrote a Python program to generate the schedules for me. I put in various constraints (when people are unavailable, what days they prefer, etc.) and then the program uses a hill-climbing algorithm along with a bunch of scoring routines to find a good schedule. The actual workload operates on Schedule objects, which just contain a list of dates and the members assigned to each position on each date. Then I make a ton of duplicates, each with one adjustment, score them all, pick out the best, repeat. I can also optionally go for two adjustments, which takes much longer but gives better results, and that's what ends up taking 10+ minutes as above.
Things seem adequately speedy, haven't investigated the network throughput tweaks yet.
[update] https://travis-ci.org/gevent/gevent passes w/ pypy fwiw.
Some projects uses Swig precisely because of that characteristic. Project with "low bandwidth" for whom maintaining different binding generator for every language they support would be painful.
In my case, I could probably hack a minimal cffi binding. But solving the problem at the root would be a better solution for everyone.
Swig already use some form of cffi for Common Lisp apparently so maybe python-cffi support could be derived from that. I don't know for sure, just thinking out loud.
That way, new projects can still benefit from the strength of SWIG that allows them to propose bindings for multiple languages with a single tool. Because, let's not kid ourselves, using specific binding libs for every single language you want to support has a cost. And in that case backward compatibility does not matter much.
All the while giving a transition path for projects still using what I would call the legacy way. Or those who want/need to poke in the CPython API.
contents = open(file).read()
where they should be doing
with open(file) as fid:
contents = fid.read()
f = open('foo.txt')
contents = f.read()
with open('foo.txt') as f:
contents = f.read()
f = open('foo.txt')
contents = f.read()
Technically the guarantee in CPython is "soft" so it can be a problem, just rarely is: if the file gets referenced from an object accessible from a cycle the file won't be released (and closed) until the cycle-breaking GC is triggered.