
A Python Interpreter Written in Rust - rch
https://github.com/RustPython/RustPython
======
hathawsh
This is wonderful. This could become the best way to move Python projects to
Rust: initially just run on the RustPython interpreter, but then optimize low
level routines in Rust. In 15 years I wouldn't be surprised if this or
something like it surpasses CPython in popularity.

Still, no discussion about Python implementations is complete without some
mention of the infamous GIL (global interpreter lock). :-) CPython has it,
Pypy doesn't (I think) (EDIT: yes it does), Jython doesn't, etc. What is the
GIL plan for RustPython?

~~~
amelius
Writing a concurrent runtime system including garbage collector is a serious
effort, and that's why all those other versions of Python don't support it and
are stuck with a GIL. Hence, I highly doubt that this Rust version of Python
has gotten rid of the GIL.

I'd love to see a better separation of language and VMs. I think it's a bit
sad that a language designer has to either implement their runtime system from
scratch, or has to run it on top of a VM that was designed for another
language (Java in the case of Jython).

Therefore, the thing I'm looking forward to most is a concurrent, generic and
portable VM written in Rust.

~~~
pdpi
> I'd love to see a better separation of language and VMs. I think it's a bit
> sad that a language designer has to either implement their runtime system
> from scratch, or has to run it on top of a VM that was designed for another
> language (Java in the case of Jython).

Wasn't Perl 6's Parrot kind of meant to fulfil that role?

~~~
chromatic
> Wasn't Perl 6's Parrot kind of meant to fulfil that role?

Yes, that was an original project goal. You can see this as far back as
Larry's State of the Onion 2003:

[https://www.perl.com/pub/2003/07/16/soto2003.html/](https://www.perl.com/pub/2003/07/16/soto2003.html/)

... the "Parrot: Some Assembly Required" article written by Simon Cozens in
September 2001:

[https://www.perl.com/pub/2001/09/18/parrot.html/](https://www.perl.com/pub/2001/09/18/parrot.html/)

... or, if you trust Git commits rather than articles which could have been
edited in the meantime, the same article revised as introductory docs in the
Parrot repository in December 2001:

[https://github.com/parrot/parrot/blob/9bc8687beb5180e4cc8971...](https://github.com/parrot/parrot/blob/9bc8687beb5180e4cc89712dc69d0b828670d5ba/docs/intro.pod)

~~~
lizmat
I stand corrected.

------
lykr0n
Neat. Was able to clone the repo, run cargo run, and drop into a python shell.
Doesn't seem like can do much right now, but I really like the idea.

    
    
      >>>>> a = [1,2,3]
      >>>>> a[2:]
      [3]
      >>>>> a[1:]
      [2, 3]
      >>>>> fh = open('~/.ssh/id_rsa.pub', 'r')
      thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: RefCell { value: [PyObj instance] }', src/libcore/result.rs:999:5
      note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
    

What would be really cool if this could one day be like Nuitka- but in rust.
Write in python, compile into Rust. Maybe even support inline Rust like
cPython supports inline C.

~~~
Ralfp
> cPython supports inline C

First time I am hearing this. Can you share an example?

~~~
lykr0n
[https://cffi.readthedocs.io/en/latest/](https://cffi.readthedocs.io/en/latest/)
&
[https://docs.python.org/3/extending/extending.html](https://docs.python.org/3/extending/extending.html)

There also is [https://github.com/rochacbruno/rust-python-
example](https://github.com/rochacbruno/rust-python-example) which is
something I want to look into.

As for an example, I'm using Snappy right now, so here you go:
[https://github.com/andrix/python-
snappy/blob/master/snappy/s...](https://github.com/andrix/python-
snappy/blob/master/snappy/snappy_cffi_builder.py) &
[https://github.com/andrix/python-
snappy/blob/master/snappy/s...](https://github.com/andrix/python-
snappy/blob/master/snappy/snappy_cffi.py)

It still requires you to compile the C part- which is why sometimes you need
GCC when you're doing a pip install.

~~~
stochastic_monk
I’ve done a lot of interfacing with C and C++. pybind11 [0] has been the
easiest and most effective for me to use. It targets C++, but it’s easy enough
to wrap C code with it. Cython brings along a lot more bookkeeping code and
overhead in my experience. cffi isn’t bad, but it’s not as
flexible/expressive.

[0] [https://github.com/pybind/pybind11](https://github.com/pybind/pybind11)

~~~
filmor
CFFI has the huge advantage that you don't need to link against libpython,
which in turn allows you to have a single wheel for multiple Python versions.

------
Animats
It's a naive interpreter. The source is parsed into a tree and then flattened
into byte code. All values are stored in PyObjectPayload structures. Each
operation has a function. Here's "hex":

    
    
        fn builtin_hex(vm: &mut VirtualMachine, args: PyFuncArgs) -> PyResult {
            arg_check!(vm, args, required = [(number, Some(vm.ctx.int_type()))]);
    
            let n = objint::get_value(number);
            let s = if n.is_negative() {
                format!("-0x{:x}", n.abs())
            } else {
                format!("0x{:x}", n)
            };
    
            Ok(vm.new_str(s))
        }
    

There's a big dispatch table, generated at compile time, and an interpreter
loop. Just like you'd expect. It's useful if you happen to need a Python
implementation to embed in something and want something cleaner than CPython.
Probably slower, though.

------
4gotunameagain
Serious question with no irony at all. But why? How is this useful?

~~~
pm90
Serious Question: does everything need to be useful?

~~~
blocked_again
I think everything in the Universe is useful for some purpose. Can you name
anything that is completely useless?

~~~
saghm
Apparently the word "useless" itself, from what you're saying

~~~
blocked_again
If word "useless" is actually useless then the word "useless" cannot be
useless

~~~
saghm
Oops, Russell's paradox strikes again

------
orf
If anyone is at FOSDEM this will be presented tomorrow in the rust room. Can't
wait!

~~~
ZeikJT
I found the slides and info at [1] but does anyone have a recording of this
presentation?

[1]
[https://fosdem.org/2019/schedule/event/rust_python/](https://fosdem.org/2019/schedule/event/rust_python/)

~~~
ephesee
The video is now available on the page.

------
sametmax
If anybody had any doubt about how welcome the project is...

Since this has been posted to HN, the repo got 8 new PR.

------
gonational
Now this is something I will donate money to…

Python with Rust as its foundation sounds like the best idea ever.

I’m curious to know whether or not it would be possible (or reasonable) to
eventually get the same or better performance as CPython.

~~~
Waterluvian
CPython is really quite slow by design. The reference implementation is meant
to be obvious and pretty easy to interoperate with C.

I think a RustPython implementation would be pretty cool. You could definitely
take that opportunity to worry about performance more than CPython does while
also worrying about interoperability more than PyPy does.

Or I'm missing your point and you're suggesting a drop-in replacement for
CPython that supports _all_ the same C-based libraries as CPython does.

~~~
Twirrim
PyPy is doing really well on compatibility these days. It can use all the C
libraries natively, and a lot of the speed issues there have been resolved.

~~~
chrisseaton
> It can use all the C libraries natively

Literally all of them, without any issues? I'm working on implementing support
for Ruby's C extensions in an alternative implementation and it's a right
slog.

~~~
mattip
PyPy dev here. Yes to both - all of them and it was a right slog. If it
doesn’t work report an issue. We have a proposal looking for funding to make
it fast.

~~~
Waterluvian
This is wonderful news to me. So broadly speaking I should be able to drop in
replace cpython with pypy for my fairly not so special projects?

I have a Django app that does some heavy data serialization and I'm not yet
ready to optimize those serializers in another language.

I can't wait to try this out.

Crumb. I didn't realise pypy is on 3.5.3. Loves me my f strings.

~~~
mattip
We backported fstrings to 3.5. We also have an alpha quality linux 3.6
available on nightly builds

~~~
Waterluvian
Great! I appreciate you sharing this. I checked the PyPy main page, Downloads
page, and Compatibility page and while I didn't look exhaustively, nothing
mentioned 3.6 or 3.6 features.

------
systems
I wish, I wish, I wish .. something like this is done for Tcl/Tk

I like Tcl the language a lot, and I love the idea of two language systems

One high level for scripting Tcl One low level for high performance commands
and parts Rust

You can of course do that today, using Tcl and C But .. well C is no Rust

~~~
jadbox
Why do you like Tcl? I worked at a place that used it for their general
purpose web servers... and most of their programmers complained that it was
like using Fortran for modern web development. That's the only experience I've
had with Tcl professionally in the field.

~~~
systems
I think you really have to like the idea of 2 language system, to appriciate
the potential of Tcl

Having one team create complex tool using Rust, or Ocaml or Go or C++

And another team more domain oriented, created UIs and Interfaces and Script
using Tcl (a language that is simple enough , yet powerful enough)

The one size fit all language, dont exist, I think, a two language team, is
very good option

A (stretched) example, is SQL ..and the DBMS Expert programmers enrich the
DBMS using C++ or whatever And domain experts use SQL and Procedureal-SQL to
solve business problems

Tcl as a universal declarative language, is an idea I like

~~~
earenndil
> The one size fit all language, dont exist

Lisp?

(or perhaps lisp is an n language system)

~~~
systems
I think any one size fit all language, will have to have optional typing, and
be both compile and interpreted

i dont know of any language, that have both, and was successful

Not sure if clojure specs, achieve the same outcome of optional typing, so
maybe you have a point :)

~~~
earenndil
> optional typing

The problem with that is that all the languages seem either to inherently be
dynamic or static. Any attempt to add the other kind is hamstrung by the
language's inherent tendencies, and it doesn't really work. Perl or python,
for instance, have optional typing, but it's not checked at compile-time. And
then there's things like c++ or d variant that--again, they don't quite feel
quite as dynamic as they would in a dynamic language. I don't think these
features can truly coexist well in the same language.

> both compile and interpreted

That's no hard ask. You just have to fight inertia, but there's really not
much standing in the way of something like that.

~~~
dragonwriter
> The problem with that is that all the languages seem either to inherently be
> dynamic or static. Any attempt to add the other kind is hamstrung by the
> language's inherent tendencies, and it doesn't really work. Perl or python,
> for instance, have optional typing, but it's not checked at compile-time.

Python optional typing is checked in an optional pre-compilation step. Except
that there is no opportunity to use typing for optimization, this isn't
meaningfully different, when used, from being checked as part of compilation.
In fact, other than using type information for optimization it's pretty much
what most compile time type checking does; compilation isn't an indivisible
atomic step.

------
MR4D
Given all the great ideas here, maybe it’s time to fork Python. Ok, so maybe a
close derivative as opposed to a true fork.

Given the language names involved (Rust & Python), I’d like to suggest
“Copperhead” as a name for it.

~~~
sametmax
Forking implies keeping the C code.

I think a rewrite in rust is more future proof: we benefit from a safer, more
modern language to implement it, which comes with cargo, and hence, the
potential of an hybrid python/rust toolchain and dev plateform.

~~~
mlevental
copper doesn't rust (it does patina though)

------
chadrs
I stopped paying attention to the Rust parsing ecosystem for a while, curious
how LALRPOP compares to nom/pest/combine and if something about python's
grammar led to the choice.

~~~
losvedir
Totally different use cases. nom/pest/combine are parser combinators, where
you stitch the functions together yourself. LALRPOP is more in the vein of
yacc where you specify the _grammar_ and it generates the Rust parsing code
for you in a build step.

~~~
surajrmal
pest isn't a parser combinator. It also uses a grammar file.

~~~
losvedir
Oh, wow! Thanks for the correction. I had somehow missed that. I'm working on
a rust parser of Elixir myself, and using LALRPOP for it. But looks like I'll
have to check out pest, too. LALRPOP has the advantage that it takes a BNF
grammar, which the Elixir parser in erlang includes, so it's a somewhat
straightforward translation. Not too sure how similar it is to a PEG grammar
that pest takes.

~~~
ltratt
LALRPOP is an LR parser, which is a very different formalism to PEG: it's easy
to write a grammar in either that's difficult/impossible to express in the
other.

If you'll permit the immodesty, another Rust parser is lrpar
([https://crates.io/crates/lrpar](https://crates.io/crates/lrpar)) which is a
more direct drop-in replacement for Yacc, but with better error recovery.
[Note: I'm biased because I wrote parts of lrpar and the wider framework,
grmtools, it's a part of.]

------
cs702
Yes. In particular, I like the potential for:

* using Rust's borrow-checking to develop new lightweight/shared-memory multiprocessing tools for Python (think "import SharedMemoryPool from multiprocessing") without having to mess with the GIL, so as to maintain compatibility with existing libraries;

* using Rust's type inference on Python code for applications in which type safety is highly desirable; and

* compiling Python code for speed, targeting all architectures and platforms supported by Rust (e.g., WebAssembly).

~~~
mhh__
How does an interpreter written in a statically compiled language use the type
system of said language after being compiled?

~~~
cs702
Not sure. Perhaps this project could eventually lead to compiled Python...?

------
yuchi
I want to create the Ramon’s Law (as a corollary of Atwood’s Law):

> Anything that can be Written in Rust, will Eventually be Written in Rust

Please go on, cite me, consider it «Attribution, Share-alike»

~~~
sramsay
Maybe, "Anything that can be Written in Rust Will Eventually be Suggested, and
then Celebrated with Great Fanfare on HN as the Obvious Future of Reality."

Atwood's Law was a prediction about the future that was based on something
that had _already_ happened (zillions of apps and libraries rewritten in JS).
I'll quote your law when we have anything like comparable evidence that this
is happening with Rust.

~~~
yuchi
Fair enough.

------
amelius
Cool. Rust with a garbage collector :)

------
nostrademons
Curious what the general story on Rust <=> Python interop is like now (beyond
writing a Python interpreter in Rust). I'd checked out rust-cpython a couple
years ago, and I see there's now PyO3, but I just looked at some code samples
and it seems like they're a long way away from the convenience that you get
with CFFI + C code, boost::python, or even SWIG. Anyone have experience
writing Python extensions in Rust or embedding a Python interpreter into a
Rust program? Could you comment on how hard the process was and how robust the
result is?

------
bibyte
This is awesome. I hope more features from CPython are ported. It looks a
little bare bones right now. Is there any chance it will gain full Python
compatibility ?

~~~
FartyMcFarter
Let's wait for it to support dictionaries first, before we start dreaming
about full compatibility.

edit - it looks like it supports dictionaries with string keys, but not with
integer keys.

------
FrankDixon
How nice, even a wasm interpreter, that's dandy

------
kodablah
Neat. Now keep a hash of the types/info for instructions at runtime, and when
a set is called enough times and appears reasonably pure/simple, JIT it with
cranelift (akin to a tracing JIT).

------
tyingq
Is there anything about Rust that would make it hard to use existing python
extensions that are written in C? I see the FFI docs, so it seems like it
would be fine, but have no Rust experience.

~~~
biesnecker
Fwiw, nothing about rust (probably) but a whole lot about CPython's
implementation. Extensions can and do poke around in object internals and will
break if the internals don't work the way the extension expects them to (that
is, the way CPython works). This has long been PyPy's big stumbling block
(though it seems to have gotten better).

------
tasubotadas
Does it have a GIL? If not, it's already better than CPython :-D

------
fxfan
I don't get the wasm craze in the rust crowd. There is no popular language
that cannot be compiled to wasm. It just feels sad to spend time there.

~~~
mjw1007
If this ended up making it possible to run Python in the web browser, that
might be interesting to a fair number of people.

(There are advantages to running the same language on the server and client,
and there's plenty of server-side web application Python code out there.)

~~~
soperj
>There are advantages to running the same language on the server and client

I see this said quite often. What are they?

~~~
coldtea
Not having to learn 2 languages and duplicate efforts for starters...

------
Thaxll
Another amateur project that will go no where, look back at this project
project 5 years from now, nothing will run on it / be maintain.

