This is wonderful. This could become the best way to move Python projects to Rust: initially just run on the RustPython interpreter, but then optimize low level routines in Rust. In 15 years I wouldn't be surprised if this or something like it surpasses CPython in popularity.
Still, no discussion about Python implementations is complete without some mention of the infamous GIL (global interpreter lock). :-) CPython has it, Pypy doesn't (I think) (EDIT: yes it does), Jython doesn't, etc. What is the GIL plan for RustPython?
The GIL comes with great convenience as you don't have to worry about a whole host of data races. It's no silver bullet but it's mighty convenient if you only do some multi-threading like in a web-server.
Many libraries are not prepared for the disappearance of the GIL and while it's not a general problem for python per se it will be a great amount of work to make every library compatible with GILless python.
Therefore I think that you must always provide an option for the GIL that is enabled by default in order to provide backward compatibility.
> The GIL comes with great convenience as you don't have to worry about a whole host of data races.
This is true, but it doesn't mean that a GIL-less Python would need to have an option for "enable GIL so I don't have to worry about data races". It means that a GIL-less Python would have to ensure that there are no data races, without having to have a GIL.
> it will be a great amount of work to make every library compatible with GILless python
No, it won't; libraries won't have to change at all. The current interpreter makes a guarantee that those libraries rely on: "you don't have to worry about data races". A GIL-less interpreter would have to make the same guarantee; it just wouldn't have to have a GIL to do it. That requirement is what makes a GIL-less Python interpreter hard to do.
Right now you have one mutex for everything (the GIL itself) and everything else doesn't need locking. In order to achieve similar convenience without the GIL you would have to trade this for one mutex for every single data structure. Because the data structures in python are so flexible, every single variable needs its own mutex then. Locking every single variable access would be enormously costly.
Other languages achieve a good compromise by clustering data structures into fewer parts with only a handful of mutexes that are locked while other threads work on different datasets. This is usually done manually and with great care as it is the heart of both safety and performance. I don't know if there is an automatic solution to this problem that is compatible with the way python programs are structured.
The libraries basically assume that, while you call them, nothing else changes. In order to ensure that you need to lock everything down. Because you don't know what these libraries do and what data they access it needs to be everything (like it is today). It should be possible to only lock the GIL when such a library is called, so there should be kind of a middle way forward.
> Right now you have one mutex for everything (the GIL itself) and everything else doesn't need locking.
If this were true, all of the explicit locking mechanisms in Python's threading module would be pointless. But in fact the GIL's "mutex" is quite a bit more limited than you are saying. It does not prevent all concurrent code from running. It only prevents Python bytecode from running concurrently in more than one thread. But the GIL allows switching between threads in between individual bytecodes, and "one Python bytecode" does not correspond to "one Python statement that performs an operation that you want to be atomic"; plenty of Python expressions and statements are implemented by multiple bytecodes, so it is perfectly possible for multiple threads executing concurrently to modify the same data structures with such statements, creating race conditions if explicit locking mechanisms are not used to prevent it. That's why Python's standard library provides such explicit mechanisms.
> If this were true, all of the explicit locking mechanisms in Python's threading module would be pointless.
Not true. You can have serialized access to the same data structure that still have data race.
But as long as each Python process doesn't keep its local copies for those shared data structures, like free lists, no explicit locking is required if GIL is presented.
> You can have serialized access to the same data structure that still have data race
How?
> as long as each Python process doesn't keep its local copies for those shared data structures, like free lists, no explicit locking is required if GIL is presented.
I have no idea what you're talking about. Different Python processes each have their own GIL, and they don't share data at all (except by explicit mechanisms like communicating through sockets or pipes). Different Python threads share the GIL for their interpreter process, and if each thread doesn't keep its own local copy of data, there is explicit locking required if you don't want data races.
Simplest scenario, the read-increment-write cycle with 2 threads. Even with a mutex, it is still possible to have data race, if the lock is on per operation level.
For the second part, yep, it is a mistake, not processes, but threads.
With GIL, the thread is given the permission to operate on certain interpreter-related data-structures, like reference counts, or free_list like in PyIntObject. What I mean the active thread is free to modify those data structures without fear of data races, and there is no explicit locking required, if it doesn't hold its own copies of those interpreter internal states.
But GIL can only guard the interpreter's own states, not any user program's states. And yes, explicit locking for operating on your own data is still required.
> Simplest scenario, the read-increment-write cycle with 2 threads. Even with a mutex, it is still possible to have data race, if the lock is on per operation level.
What you're describing is not "serialized access with a data race"; it's "multi-thread access that you didn't explicitly control properly".
> For the second part, yep, it is a mistake, not processes, but threads.
Ok, that clarifies things.
> the active thread is free to modify those data structures without fear of data races, and there is no explicit locking required, if it doesn't hold its own copies of those interpreter internal states.
I'm not sure I see why a thread would want to hold copies of those interpreter internal states, since if it did the issue would not be modifying them properly but having the local copies get out of sync with the interpreter's copies, since other threads can also mutate the latter.
The problem with GIL is that it's a mutex that you don't control. So you can't use it to do atomic updates, if that involves more than one low-level operation that is atomic.
So in practice I don't think it simplifies things all that much. If anything, it creates a false sense of security - first developers get used to the fact that they can just assign to variables without synchronization, and then they forget that they still need to synchronize when they need to assign to more than one atomically.
I'm pretty rusty on Python but my impression wasn't that the GIL meant just "no data races" but that it also meant "data can't change out from under you in the middle of executing a statement". You could write a Python interpreter that ensured no data races and yet still had divergent behavior by allowing shared data to be mutated by another thread halfway through executing a statement.
> my impression wasn't that the GIL meant just "no data races" but that it also meant "data can't change out from under you in the middle of executing a statement".
That's not quite what the GIL guarantees. It guarantees that data can't change out from under you in the middle of executing a bytecode. But many Python statements (and expressions) do not correspond to single bytecodes.
Writing a concurrent runtime system including garbage collector is a serious effort, and that's why all those other versions of Python don't support it and are stuck with a GIL. Hence, I highly doubt that this Rust version of Python has gotten rid of the GIL.
I'd love to see a better separation of language and VMs. I think it's a bit sad that a language designer has to either implement their runtime system from scratch, or has to run it on top of a VM that was designed for another language (Java in the case of Jython).
Therefore, the thing I'm looking forward to most is a concurrent, generic and portable VM written in Rust.
I think there's an analogy between the two issues you brought up.
1. A concurrent garbage collector is 10x more work than a single-threaded one. People often don't realize this.
2. A language-independent VM is 10x more work than a VM for a given language. People often don't realize this this.
In other words, VMs are tightly coupled to the language they implement, unless you make heroic efforts to ensure otherwise.
WebAssembly is a good example of #2. I think the team is doing a great job, but they are inevitably caught between the constraints of different languages (GC, exceptions, etc.)
The recent submission The Early History of F# sheds some light on this with respect to the CLR:
An outreach project called “Project 7” was initiated: the aim was to bring seven commercial languages and seven academic languages to target Lightning at launch. While
in some ways this was a marketing activity, there was also serious belief and intent. For help with defining the academic languages James Plamondon turned to Microsoft Research (MSR).
I think this is the only way to design a language-independent VM -- port a whole bunch of languages to it. And there are probably 4 or 5 companies in the world with the resources to do this.
I've seen some VM designs that aim to be generic, but since they were never tested, the authors are mistaken about the range of languages they could efficiently support.
Of course, you can always make a language run on a given VM, but making it run efficiently is the main problem.
> I'd love to see a better separation of language and VMs. I think it's a bit sad that a language designer has to either implement their runtime system from scratch, or has to run it on top of a VM that was designed for another language (Java in the case of Jython).
Wasn't Perl 6's Parrot kind of meant to fulfil that role?
... or, if you trust Git commits rather than articles which could have been edited in the meantime, the same article revised as introductory docs in the Parrot repository in December 2001:
I'd say not originally. But over time, as the Perl 6 project got delayed in the 2000's, it was decided that Parrot would be a runtime for all scripting languages. Which in turn meant it couldn't cater well enough for any. Which led to its demise.
Having recently been down this particular rabbit hole myself; I just want to note that there are other possible strategies, a GIL is not the only alternative to a fully concurrent runtime.
My own baby, Snigl [0]; doesn't even support preemptive threads, only cooperative multitasking; with a twist, since blocking operations are delegated to a background thread pool while yielding when multitasking.
That is a nice strategy but it only allows IO to be parallel, and leaves the CPU sequential. Programs may be concurrent (because of the cooperative multitasking) but not parallel (because there is only one CPU thread). Users may find it disappointing that they can only use a single core.
Also, keep in mind that cooperative multitasking may cause unexpected high latencies, which is unfortunate e.g. in GUI applications and web servers; this is a result of queueing theory, and an example is given here: https://www.johndcook.com/blog/2008/10/21/what-happens-when-...
I am aware, it's simply the most portable universal solution to the problem that I could think of. AIO comes with its own share of issues; Unix (and thus Posix) are pretty much dead; and it's not universal, I can't use it to open a database connection unless the database has support built-in.
From what I know, cooperative multitasking suffers significantly less from unpredictable performance than preemptive threading. The biggest source of uncertainty in Snigl is the preemptive IO loop.
The things is that I really don't feel like writing a concurrent runtime; been there, done that. I'm planning something along the lines of Erlang's processes and channels based on separate interpreters for those scenarios.
Ok. A problem with separate processes is that you have to serialize your data when passing messages. This is a pity because functional data structures + structural sharing can be very efficient. For example, I can't imagine how someone would implement a high performance database without structural sharing.
They're implemented (in my mind so far) as preemptive threads, one per interpreter; which makes them slightly more heavy-weight than Erlang's NxM and a nice complement to purely cooperative multitasking.
It's one of those languages that does things differently to solve actual issues, not to check boxes.
From my limited experience, Erlang doesn't share data between processes; you throw it over the fence by sending to the process inbox, which is where the locking takes place.
Still, shuffling data between OS threads is an easier problem to solve than serializing between OS processes.
I am working on something similar but from a slightly different direction. The project is mainly focusing on compiler infrastructure for now, but I have a reference interpreter I use for validation. The short term goal is to make it able to run the erlang `compile` module. https://github.com/hansihe/core_erlang
Compatibility with C extensions seems to be the most difficult thing for an alternative implementation to achieve. PyPy struggled with this for a long time, and IIRC extension compatibility also caused the failure of Dropbox's Pyston.
Is there really no situation in which an alternative implementation that only supports "pure Python" would be useful?
> Is there really no situation in which an alternative implementation that only supports "pure Python" would be useful?
This really depends on your definition of 'being useful'. Jython is useful in a sense, it is being used in many Big Data solutions as a way to embed Python as DSL/UDF, like Pig/Hive, etc.
However, if without support for C extensions, it is not really a Python implementation, in a sense, I can't run a python script I just gripped from internet using the so-called 'alternative' implementation. So if the point of being useful is to be a replacement, then sadly, the answer is no, it is an everything-or-nothing situation.
In the old days, before Eclipse had a scratchpad area or Java finally got a shell, it was useful to me as means to explore some Java APIs, as I did not want to spend time with either Beanshell or Grovvy since I know Python superficially since 1.6.
Rust + Python seems a natural combinaison to me, and being able to have one single dev env (and maybe in the end, one single deployment mechanism) to do both is a killer feature.
And actually, I think having Python written in Rust would provide some other very nice properties:
- limit the number of bug you can introduce in the implementation because of the rust safety nets;
- can still expose a C compatible ABI and hence be compatible with existing extensions;
- the rust toolchain being awesome, it may inspire people to make it easy to compile a python program. Right now I use nuikta, which is great, but has to convert to C then compile, which make it a complex toolchain.
> This is wonderful. This could become the best way to move Python projects to Rust: initially just run on the RustPython interpreter, but then optimize low level routines in Rust. In 15 years I wouldn't be surprised if this or something like it surpasses CPython in popularity.
What you are describing is simply a JIT compiler. Maybe are you suggesting to rewrite PyPy (its C part) in Rust?
Actually I was referring to manual translation of Python code to Rust. My experience with PyPy has been rather bad, unfortunately (it runs my code at less than half the speed of CPython) and I figure human translation should be a lot more effective than JIT optimization.
Maybe you should send your code to the PyPy team. I bet they will look into it and see how they could improve their JIT compiler. Last time I checked the project, they had a full benchmark running for each release to spot regressions and to track improvements
"What you are describing is simply a JIT compiler."
The user is describing the opposite of a JIT compiler: a gradual rewrite of Python apps in Rust to feed into an ahead-of-time, highly-optimizing compiler. A JIT compiler would do quick compiles of Python code while it's running. The performance, reliability, and security capabilities of JIT vs AOT vary considerably with context. For predictability and security, I avoid JIT's wherever possible in favor of AOT's.
Grumpy was supposed to accomplish the same for Python->Go, and although now abandoned, probably holds some lessons in how to design a platform to help Python projects get Rusted.
Grumpy compiled python code to fairly unreadable Go, and then quickly compiled the result. One effect of this is that a programmer could theoretically refactor the resulting Go code gradually.
I would personally try to move away from python at this point for greenfield projects. The GIL is so baked into the language, if you removed it a bunch of current python code will probably break in subtle ways.
Modern languages need proper multithreading support, static types and fast compile speeds. Use golang, use kotlin, use dart, use anything but python & javascript.
It's basically impossible to implement a CPython-compatible language without a GIL (or you lose single thread performance by using very fine-grained atomics/locking). Python has very specific multithreading semantics that are a function of the CPython bytecode and the GIL, and programs rely on this.
Other than refcounting (which is not a part of the Python language spec - it even specifically says that conforming code shouldn't rely on it), what other semantics did you have in mind?
Guarantee or not, it constrains whether something is usable as a drop-in replacement interpreter, especially if people can't tell which programs will break, and doubly so if the breakage is a subtle data corruption race that doesn't show up in tests.
I want it to have the GIL, because I want it to maintain compat with C extensions.
We already have a plan to bypass the GIL: multi interpreters.
Having an implementation in Rust may make future improvement to Python easier, so it's better to have something exactly similar first, then start to hack it.
Still, no discussion about Python implementations is complete without some mention of the infamous GIL (global interpreter lock). :-) CPython has it, Pypy doesn't (I think) (EDIT: yes it does), Jython doesn't, etc. What is the GIL plan for RustPython?