I agree. I think there needs to be an effort to push towards concurrency in Pypy. They were optimistic that the GIL would be easier to factor out in their runtime, and I think that could be a great start. I'm all for new methods for concurrency, but I feel being able to make efficient use of native threads would bring a lot of momentum to the project. Even though it's not as big a deal as most people make it out to be, Python gets a bad rap for the well publicized GIL problems.
Everyone is optimistic that they can factor out the GIL (see: Unladen Swallow). Then they find out the problem is much, much harder then they initially thought. That said, all it requires is a fundamental change (e.g. dropping) the ref counting system:
So, perhaps PyPy can do it, and I hope they do - from an evolutionary standpoint, they're the interpreter with the most velocity inside of the interpreter core development, so if it's going to happen, it's going to happen there.
CPython will stay conservative, and much as it is today for some time.
For a while I had it in my mind that maybe you could save some memory while having safety just by using os.fork() after pre-importing a bunch of stuff; but then reference counting was pointed out to me and I realized it would be pointless -- the act of merely looking at an object changes the object, so copy-on-write means copy-everything.
Reference counting seems like it could be removed... except for all the C extensions that are specifically handling reference counting internally, removal of which I'm guessing would cause a lot of breakage. But then... it seems like those C extensions are exactly the problem, so if PyPy tries to keep them is it messing itself up? (Though I'm guessing it would somehow use proxy objects or some other trick to keep its internal object representation flexible.)
Ah! But you see, it's possible to trick CPython extensions into thinking you do support refcounting: http://code.google.com/p/ironclad/ is proof. So you can be compatible, yet advanced :)
> ...reference counting was pointed out to me and I realized it would be pointless -- the act of merely looking at an object changes the object, so copy-on-write means copy-everything.
Hmm...might it be worthwhile to work around this by storing reference counts in a separate set of pages, and having each object instead hold a constant pointer to its refcount? Presumably the extra level of indirection would have some degree of performance impact, but if it allows CoW to actually work in a meaningful way...
Though I guess that would probably break every C extension in existence, no? (Note: I know absolutely nothing about Python's C extension API, just guessing.)
