I don't understand how it's possible that we just randomly come across a project that just casually implements a Python interpreter in Rust. Don't these things take a massive amount of effort? Wouldn't this be making waves much earlier in its development process?
I feel the same way about Ruff, for example. One day it was "black all the things" and the next it's "btw we just reimplemented the entire Python formatting/linting ecosystem in Rust, and it's 100x faster, no biggie".
What's happening? Is it just so much easier to write stuff in Rust that projects like these pop out of people's heads, fully-formed? It boggles the mind.
For me the "uv" manager has changed my Python experience because: (1) it has a correct resolver whereas "pip" certainly doesn't and I'm not sure about poetry, (2) it is crazy fast, (3) "uv" is just a binary which I can pip into my system.
(3) is important because if it was written in Javascript or Java or Python or .NET or many other languages I'd have to learn something about the runtimes of those environment to get it working. If it was written in Python it would have to deal with the bootstrapping problem that it ought to have it's own Python installation separate from the one that it is manipulating so it can't have conflicts with that environment. (e.g. how many times have I busted my poetry?) I can use "uv" or "ruff" without learning anything about Rust!
As for (2) the speed of "uv" has as much to do with better algorithms and caching as it does with being in Rust and thus much faster than Python. I think you could have done better than Poetry in Python but "uv" is transformative in that it can often build an environment in seconds or less whereas with "poetry" or "pip" or "conda" I might have time to pound out a few posts on HN. I used to avoid creating new Python environments as much as possible but now it is fast, easy, and even fun.
I bet it is more work to write "uv" in rust as opposed to a similar tool in Python but the impact on the community is so huge because we can finally put problem (1) behind us and do it with speed, reliability and grace. I had notes on how to build a better python package management system and sometimes thought about trying it but I'd become convinced that the social problem of too many people finding half-baked tools like "pip" and "poetry" acceptable was intractable. Thanks to "uv" nobody will ever have to write one.
I'm really looking forward to uv being a drop-in replacement for Poetry. I don't know if that's what they're planning to do, though. Does it currently have all the niceties of Poetry (dependency management, locks, building wheels, etc?).
"Rye supports two systems to manage dependencies: uv and pip-tools. It currently defaults to uv"
I've been evaluating it lately and it has pretty much the same CLI commands as Poetry except it's faster and comes with complete Python interpreter management (which is to me the real killer feature as I don't really care about speed of dependency resolution, but I do care about the DX).
Yeah, that's definitely within scope for what we're trying to build, and we've been hard at work on extending uv to support those workflows (platform-agnostic resolution, lockfiles, etc.). Honestly, a lot of it is already implemented, but not yet stabilized or announced. Coming soon.
Implementing a interpreter like that isn't as hard as you probably think as the standard library does a lot of the heavy lifting once you have the basics.
It's still a lot of work but the only need to make the "built in" parts of the language and that's a lot smaller subset.
I've had some fun converting some of my Python scripts into Rust and it's really not that difficult with the help of modern tools once you wrap your head around Rust. Python is too huge to crank out in an afternoon, for sure, but on the human level, the translation from python to something compiled is a well trod path.
A quick check on the contributors page shows ~8ish heavy contributors working over the course of 6 years and 13k commits. That's a good thing to check for any project you're thinking about integrating with IMO.
That said, my experience has been that adding business features in Rust apps is quite fast indeed!
Interesting that it relies on OpenSSL, either dynamically from the OS or vendored at compile time. I wonder what the implications would be for using something like rustls. You’d get TLS batteries included and kill a large external dependency… but possibly introduce behavior changes to low-level cryptographic operations, which is scary.
Still, the maintainers stated that they don’t plan to implement Python’s readline module because they already have a rust implementation of readline. A similar argument could apply here - use native rust implementations of dependencies and expose them via the expected Python APIs. This would break some ambitious Python programs, but those probably wouldn’t consider alternative runtimes anyway.
Does numpy runs on rustpython? And other libraries used in ML (not expecting compatibility with huge libraries like torch or tensorflow, but rather, getting the leaves to work should be doable)
If not, is it at all possible to get numpy to work and other libraries written in native code? I see that rustpython also work in wasm: but what about compiling numpy's native code to wasm as well?
Regarding import cost, as it’s doing heavy IO traversing the file system, the cost heavily depends on how fast you can do IO in the hardware, and also the file system (and the OS).
So a fast SSD will help, and somewhat surprisingly putting it inside docker helps (in an HPC context, not so sure it’s implications here as we’re talking about a short scripts.)
But the context here is to port shell scripts to Python, I’m not sure how huge amounts of imports matters.
And it is probably an intrinsic problem of the language, unless we start talking about compiling (and somehow statically) not interpreting the Python program, whichever implementation of the language probably won’t help the situation.
Lastly, if high startup costs of the script becomes relevant, perhaps it is orchestrating wrong. This is an infamous problem of Julia, and their practice is then just keep the Julia instance alive and use it as “the shell”. Similarly, you can do so in Python. Ie rather than calling a script from the shell acting on millions of things, write a wrapper script that start the Python instance once. Memory leak could be a problem if it or its dependencies are not well written but even in that case you have ways to deal with that.
Well CPython has a lot of backwards compatibility to deal with that RustPython doesn't, so "import subprocess" might result in very different behavior.
I wonder if this would make Python web applications more secure at interpreter and library level.
Running it on hardened Linux, OpenBSD, or FreeBSD was a start. A Rust implementation might help.
I also miss setups like eCos RTOS where a GUI determined which features got compiled in. Strip each Python app down to just what it needs in the interpreter. Might squeeze it in L1-L2 cache that way, too. Aside from embedded (eg MicroPython), has anyone anything like that for use on servers?
This seems very weird to me. Anyone who is just slightly interested in the project would want to know if specs are fully implemented and this has parity with the "official implementation". Can't believe it's not in README.
I wonder why you stops there midway in a sentence and only after reading the other comments I get what you mean …
What problems you are referring to exactly, and how would they be fixed as an implementation but not at the language level?
They list the repository as MIT licensed, but the python modules are distributed under the Python Software License which says:
2. Subject to the terms and conditions of this License Agreement, PSF hereby
grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce,
analyze, test, perform and/or display publicly, prepare derivative works,
distribute, and otherwise use Python alone or in any derivative version,
provided, however, that PSF's License Agreement and PSF's notice of copyright,
i.e., "Copyright (c) 2001-2024 Python Software Foundation; All Rights Reserved"
are retained in Python alone or in any derivative version prepared by Licensee.
The complete semantics of Python object lifetime are expensive to implement in a compatible manner without a GIL. Jython got around this by not doing it, making it not fully compatible (yes, people do depend on objects being eagerly freed), just using the JVM GC instead. If you do want full compatibility, the choice is between single-threaded performance and parallelism.
> yes, people do depend on objects being eagerly freed
I get that this must be one aspect of the necessity of the GIL but I mean, C++ also has eager free behavior due to RAII and threads are working fine there, as long as you know what you're doing. Perhaps that's the rub though, it's pretty easy to crash/deadlock in C++ and we blame the programmer rather than the language.
Idiomatic C++ relies much more heavily on ownership and not so much on refcounting. If you have code that's a rat's nest of shared_ptr, it's going to perform very poorly in a multithreaded environment. But that's why any C++ guru will tell you to not make a rat's nest of shared_ptr. When refcounting is commonly used in C++, like with GUI code or dependency graphs of network requests, it's usually in non-performance-critical sections.
In Python, by contrast, all variables default to object references, and so nearly everything you do involves updating a refcount.
Right so you're saying that Python's need to keep ref counts is what leads to the need for synchronizing updates, leading to the need for a lock, more or less. Which is only needed in C++ if you program in a kind of Python style. Makes sense and is a good point.
There is a competing c interface that numpy and a few other projects are adopting that allows for no gil. Last time this came up I thought the rust implementation used that one.
It's special-purpose, and always has been. You use Jython when you want to embed a Python-like interpreter into a Java program. Usually when you do so, you're scripting the objects of the Java program, and don't need or want to import arbitrary Python packages. Indeed, that's often the whole point of Jython - the system designer wants a language that's familiar to Python programmers, while also being able to control the environment that those Python scripts can access.
This is not different from the Python 2 days. Jython has always had subtly different semantics from Python (eg. it uses Java strings instead of Python ones, there's no C API, it relies on the Java GC so no eager free), so many common libraries wouldn't work with it. Just try to run NumPy on Jython - you can't, despite the same developer authoring both Jython and NumPy's predecessor.
I feel the same way about Ruff, for example. One day it was "black all the things" and the next it's "btw we just reimplemented the entire Python formatting/linting ecosystem in Rust, and it's 100x faster, no biggie".
What's happening? Is it just so much easier to write stuff in Rust that projects like these pop out of people's heads, fully-formed? It boggles the mind.