
Writing Python inside Rust - mcp_
https://blog.m-ou.se/writing-python-inside-rust-1/
======
6gvONxR4sf7o
I know this project may just be for fun, but with WASM targets for all sorts
of languages, I'm hoping we get to a future where mixing and matching
different languages for different parts of your program will be seamless.
Imagine starting a project in an easy language, then migrating pieces to a
faster "bare metal" language as needed in a super piecemeal way. Same with
moving pieces to a safer language as the project grows, slowly expanding the
boundaries of the safe bits as appropriate.

~~~
pjmlp
Just like the JVM and CLR I guess.

[https://en.wikipedia.org/wiki/List_of_JVM_languages](https://en.wikipedia.org/wiki/List_of_JVM_languages)

[https://en.wikipedia.org/wiki/List_of_CLI_languages](https://en.wikipedia.org/wiki/List_of_CLI_languages)

[https://www.graalvm.org/docs/](https://www.graalvm.org/docs/)

~~~
jasode
_> Just like the JVM and CLR I guess. (List_of_JVM_languages,
List_of_CLI_languages)_

I don't see JVM & CLR even with the dozens of language choices on top of the
virtual machine runtimes as giving you ways to do "bare metal" programming. I
guess one could arguably squint and say Java's JNI or C# P/Invoke and
"unsafe{}" gets you some "bare metal" capabilities but that's a stretch.

I go back to the gp's sentence: _" Imagine starting a project in an easy
language, then migrating pieces to a faster "bare metal" language as needed in
a super piecemeal way."_

For example, JVM/Java doesn't have value types[1] (yet) and that's required
for programmer's control of exact and efficient memory layout for many bare
metal domains. It doesn't matter what flavor of JVM language you use because
you're ultimately limited by the JVM's capabilities which causes excessive
pointers to pointers which is inappropriate for some high-performance code.

In C#, some future tech like CoreRT might open up some more "bare metal"
programming possibilities but that's not production yet[2]. I remember you
couldn't even develop a Windows Explorer right-click menu shell extensions in
C# because it was a bad idea to load up 2 different versions of the NET
Framework runtimes (until NET 4.0). That's not even that low a level of
programming and yet C++ had no such limitation.

I use C# as much as I can but I still have to do 50% my projects in C++
because JVM/CLR are not "bare metal" enough.

[1]
[https://en.wikipedia.org/wiki/Criticism_of_Java#Compound_val...](https://en.wikipedia.org/wiki/Criticism_of_Java#Compound_value_types)

[2] [https://github.com/dotnet/corert#user-content-net-core-
runti...](https://github.com/dotnet/corert#user-content-net-core-runtime-
corert)

~~~
pjmlp
Since when is WebAssembly bare metal?!?

Apparently you missed the Windows 8, 8.1 and 10 train regarding your "bare
metal" compilation from C# and VB.NET code.

.NET 4.0 was released in 2008 and I have been able to use C++ as .NET language
since 2001, so ....

In fact one of my first .NET projects, in 2002, was to integrate C++ RPC
library into Managed C++, long replaced replaced by C++/CLI with .NET 2.0
release.

Going back to Java, IBM, Aicas, PTC, Gemalto will happily sell you Java
compilers that generate AOT native code for embedded deployment targets, not
to mention what is running on my phone, where 95% of the OS APIs are exposed
only via Java and where your beloved C and C++ code needs to use JNI to access
them.

~~~
jasode
_> Since when is WebAssembly bare metal?!?_

Not sure what that is referring to.

 _> C++ as .NET language_

We seem to be talking about 2 different things. Using "C++" style syntax
(C++/CLI) in a _managed language with GC_ is not what many systems programmers
call "bare metal". I thought it was clear from context that my mention of C++
is traditional "real C++" such as gcc/clang/MSVC and not the C++/CLI.

 _> Going back to Java, IBM, Aicas, PTC, Gemalto will happily sell you Java
compilers that generate AOT native code_

But that's not the _JVM_ runtime though. The JVM is what you originally wrote
and that's the scope of what I was replying to: ( _" Just like the JVM and CLR
I guess. (List_of_JVM_languages ...)"_)

The _JVM_ doesn't really give you low-level bare-metal programming and it
doesn't matter what flavor of JVM-language you choose to run on top of it.

And at the risk of further muddying up the discussion with the tangent subject
of Java AOT... do any of those Java compilers give true value type semantics
or is still references with pointer chasing? I'm not familiar with those
compilers.

 _> and where your beloved C and C++ code _

"beloved"?!? Can we tone down this down a bit? I'm just trying to clarify that
JVM & CLR really don't span the entire spectrum of programming all the way
down to "bare metal" in the way low-level programmers are typically using that
phrase. I thought I was making a _neutral_ and factual statement. I.e. I'm not
interested in an emotional flamewar.

~~~
pjmlp
Maybe the tone wasn't be best one, but I would like to know how WebAssembly is
bare metal programming to start with.

Also would like to know in what way compiling natively to machine code has to
do with having value type semantics on the source language.

~~~
jasode
_> , but I would like to know how WebAssembly is bare metal programming to
start with._

We are getting into "splitting hairs" territory but let me attempt to untangle
this thread because it seems to be hung up on what _" bare metal"_ means.

Yes, if we're using "bare metal" to only mean real semiconductor chip, WASM is
not that. It's an abstract virtual machine. So yes, in that strict sense, WASM
is analogous to JVM and CLR.

But.....

I'm _charitably interpreting_ gp's comment (6gvONxR4sf7o) and he's using WASM
as his _relative_ (not absolute) perspective of _that_ being "bare metal". Ok,
if we play along with that, WASM is _not_ analogous the JVM/CLR because it is
lower level[1]. Thus a non-managed language like C++ can more easily target
WASM-flavor-of-bare-metal for high performance rather than a managed-C++/CLI
targeting .NET CLR.

Yes, it's a subtle difference. WASM is more "bare-metal-ish" than JVM ...
relatively speaking. I just don't think JVM languages can really do the same
thing as WASM as the Google/Mozilla/Apple/MS specifically engineered Web
Assembly to be a target for low-level-bare-metal languages like C/C++. In
contrast, Sun & James Gosling deliberately didn't engineer JVM Java byte code
to be a compilation target for low-level C/C++.

This means something cpu-intensive like AutoCAD or possibly Adobe Premiere Pro
can hypothetically be written to target WASM and will perform better than if
those apps were re-written in Java to target a Java web browser plugin. E.g.
Java's JVM doesn't have value types and that architecture choice is very
unfriendly to storing/manipulating millions of 3d points for a CAD program. In
contrast, WASM's architecture opens up a few more "bare-metal-ish" programming
domains.

The various choices of JVM languages like Kotlin/Clojure/JRuby/etc actually
don't address what WASM is attempting to accomplish.

[1] [https://www.quora.com/How-does-Java-bytecode-compare-to-
WASM...](https://www.quora.com/How-does-Java-bytecode-compare-to-WASM-code)

~~~
pjmlp
Yet, GraalVM compiles LLVM bitcode and WASM bytecode just fine.

[https://www.graalvm.org/docs/reference-
manual/languages/llvm...](https://www.graalvm.org/docs/reference-
manual/languages/llvm/)

[https://www.graalvm.org/docs/reference-
manual/languages/wasm...](https://www.graalvm.org/docs/reference-
manual/languages/wasm/)

A JVM and respective JIT compiler all written in Java.

And I still doesn't understand what WASM does for C++ that CLR doesn't do,
given that I can write straight C89 or C++ with C++/CLI, just like using gcc
or clang doesn't force me to use their language extensions.

------
marvy
So, at first I thought this approach is doomed, because you lose comments, and
what looks like a comment to Rust may not look like one to Python. Example:

    
    
      x = y // 2 # floor division
    

But then I decided to look at the docs:

[https://doc.rust-lang.org/proc_macro/struct.Span.html](https://doc.rust-
lang.org/proc_macro/struct.Span.html)

And I noticed source_text, which "preserves the original source code,
including spaces and comments"!!!

Why not just use this from the start then?? Seems like the easy way out, no?

(Disclaimer: I don't know Rust, can't even write hello world.)

~~~
uranusjr
The author likely just barely missed its introduction. While the article is
written recently, the implementation it talks about was published first in
early April 2019, right about when source_text was first introduced into
nightly.

~~~
ogoffart
I am the one who contributed Span::source_text to rust. My motivation for
doing that was exactly the same as the author of the blog post, and was to
help implementing the cpp! macro which embeds C++ within rust just like this
python! macro. [https://docs.rs/cpp/](https://docs.rs/cpp/) However, I still
can't make use of it today because it is not stable yet, and even proc_macro2
does not implement it (and this is not trivial to implement there)

~~~
marvy
Wow. The fact that you care about this being stable implies that you plan to
use this "for real"; the fact that you actually implemented this implies you
care enough to put in some real work. So you must have a use case in mind, and
I am not imaginative enough to guess what it might be. Is it just about
convenience so you don't need to bother with putting your C++ code in a
different file? Or is there more to it?

~~~
ogoffart
I'm using this to write Qt bindings: [https://github.com/woboq/qmetaobject-
rs/](https://github.com/woboq/qmetaobject-rs/)

> Is it just about convenience so you don't need to bother with putting your
> C++ code in a different file?

Yes, mostly. I find that having the code in place makes a big difference. I do
not like useless levels of indirection and context switches while coding. This
way is much better then having to edit three files (the .cpp, the ffi module,
and the caller) each time I want to do a call into C++ while making sure they
are in sync.

~~~
marvy
That actually makes sense! Cool!

------
oon
Using Span to get the token location is exactly what I needed! Thanks. I wrote
a blog post[0] the other day about making a css macro that compiles into rust
for the use in Yew, a front end react like framework.

> However, in rust, there’s no way to differentiate .a.b with .a .b

Now I know that the above is incorrect. I would have never thought of spans so
I thank you again

[0] [https://conradludgate.com/posts/yew-css/#what-are-the-
downsi...](https://conradludgate.com/posts/yew-css/#what-are-the-downsides-of-
yew)

------
rav
The blog post shows two Python snippets starting with "if True:" and different
indentation and says the snippets "have a different meaning". However, in this
case the difference between the snippets is mainly in their syntax and not in
their meaning. The example would have been better if "if True:" was replaced
by "if False:" or "if foo:".

    
    
            if True:
                x()
            y()
    
    
            if True:
                x()
                y()

~~~
m-ou-se
Good point! Updated.

------
ptato
What is the use case for embedding Python code in a Rust program?

~~~
choward
That's exactly what I thought when I read the title. It's very frustrating how
many landing pages, project readmes, blog posts, etc that don't answer the
question "why?". I usually need to know that before going further when I come
across something that I've never heard of before. If they don't have the
"why?" somewhere easy to find I just close the tab.

~~~
elpatoisthebest
It's a bit of a guilty pleasure for me to read these kinds of weird things
with no practical application. It's just a little refreshing to read about
something technically possibly, but frivolous (no offense meant to the OP, of
course) just because it can be done.

~~~
owl57
"Because it can be done" is a very solid answer to "Why?"

------
uijl
At FOSDEM there was a talk about boosting Python with Rust [1]. Might be
interesting for the people checking this.

[1]
[https://fosdem.org/2020/schedule/event/python2020_rust/](https://fosdem.org/2020/schedule/event/python2020_rust/)

------
pansa2
Is it common to embed one language's code directly inside another language
like this?

Lua is often used tightly-coupled to C, and it doesn't have the significant-
whitespace issues that Python shows here. Even so, I've only ever seen it used
in separate `.lua` files, never embedded directly within `.c` files.

~~~
iddan
I don’t think it’s common nor it’s a best practice as you can’t apply static
analysis on your embedded code. But it’s really cool Rust allows that and it
can be a ad-hoc solution when you need to write a piece of code fast inside of
rust

------
axegon_
While I've seen people doing this, I've never done it. I've done the opposite
in a small open source project of mine(arguably abandoned at this point due to
lack of time and contributors). That said, there is a huge potential here -
your code may require some small operation that is cheap to execute but would
take you 2 minutes to write in python while it would take you an hour of doing
the same in rust. I can think of some use cases for this. Nice article!

------
hypewatch
Why call python from Rust? I think calling rust from python would make more
sense. Use it to optimize python functions like how python libraries do with
with C (i.e. numpy)

~~~
FridgeSeal
Suppose I’m building an application in Rust that processes a lot of data, one
of the steps in processing that data I maybe want to run through a Python tool
like SpaCy or Flair.

How would I go about doing that? I could put the Python code behind a little
http API and call it that way, but that’s a bunch of overhead and extra stuff
to maintain just to analyse some text. If I embed said Python tools in my Rust
code then I can call those tools with significantly less overhead and
complexity.

~~~
kccqzy
Why does it have to have a little http API? Why not just spawn the python
tool?

~~~
FridgeSeal
And communicate with it how?

If you’re suggesting spawning things over shell/cmd line I’m of the opinion
that this a generally bad idea.

~~~
kccqzy
Just fork and exec. Or if you fancy, posix_spawn a separate process. Then
communicate over pipes, or if you have a lot of outputs from these tools,
collect results from the file system.

Really, where does this fear of spawning things come from?

~~~
jononor
Python module loading tends to be quite slow. Loading Tensorflow on our Heroku
production environment takes 10 seconds (5 on my laptop). A client of mine
running Rust and Python on Embedded Linux device found that loading
numpy/pandas modules took over 5 seconds (laptop was under 1), and their
computation took just x00 milliseconds. 10x overhead...

So the spawning of a short-lived subprocess approach has massive overhead,
only suitable for multi-second workloads.

------
ComputerGuru
I wrote this [0] up a couple of weeks ago, which is a hack to put shell code
in your rust, with reasons why you would want to.

[0]: [https://neosmart.net/blog/2020/self-compiling-rust-
code/](https://neosmart.net/blog/2020/self-compiling-rust-code/)

------
zengid
I'm having issues seeing the whole post, when I scroll down i just see white,
and when I click the subject headers from the menu it jumps down half the page
and the mouse wheel scroll gets locked. I'm using Chrome on MacOS.

------
kzrdude
Lua is a better fit for Rust, because it doesn't have the threading
restrictions of (C)Python. With Lua you can run a thousand parallel vms in the
same process if you want.

But the macro hacks are impressive!

------
amrx101
So I am not the only one with questions regarding its use case? I cant seem to
conjure any.

~~~
random32840
Quickly introducing battle-tested data science frameworks, snippets and
patterns without having to replicate them in Rust.

------
superdisk
There's a typo where stringify! is referred to as strinfigy!

Cool article btw!

~~~
m-ou-se
Thanks! Updated.

------
bsder
Am I the only person having trouble reading this blog? That color scheme makes
my eyes bleed.

It seems like it's a light gray background with a slightly (not much) darker
gray text. The contrast and thin font weight is terrible.

And the pink is practically vibrating on the page.

At least the blue is okay.

