
Grumpy: Go running Python - trotterdylan
https://opensource.googleblog.com/2017/01/grumpy-go-running-python.html
======
Animats
\- Amusingly, it runs Python 2.7, even though this project started long after
Python 3.x came out.

\- It's a hard-code compiler, not an interpreter written in Go. That implies
some restrictions, but the documentation doesn't say much about what they are.
PyPy jumps through hoops to make all of Python's self modification at run-time
features work, complicating PyPy enormously. Nobody uses that stuff in
production code, and Google apparently dumped it.

\- If Grumpy doesn't have a Global Interpreter Lock, it must have lower-level
locking. Does every built-in data structure have a lock, or does the compiler
have enough smarts to figure out what's shared across thread boundaries, or
what?

~~~
TillE
> Nobody uses that stuff in production code

Nobody uses the features of Python which make it a dynamic language? Google
must write some really weird Python if their compiler is that strict.

~~~
Animats
In Python, you can get at a variable, or even code, in another thread with
"getattr()". You can monkey-patch another thread while it is running. This is
not very useful, but it's easy to implement in a naive interpreter such as
CPython. Part of the price for this is the Global Interpreter Lock, so you
don't really have two threads running at once. PyPy has a huge amount of
machinery so that stuff will work.

Grumpy doesn't even seem to try to implement that. That's a good thing. If you
restrict Python a little, it's much easier to compile.

~~~
mdaniel
> That's a good thing. If you restrict Python a little, it's much easier to
> compile.

Isn't that more or less what RPython does?
[https://rpython.readthedocs.io/en/latest/architecture.html](https://rpython.readthedocs.io/en/latest/architecture.html)
I mean, I know that starting with a full-fledged(?) Py27 codebase rules out
_actually_ using RPython for the stated goals of Grumpy, but I think the two
projects agree in principle and differ about the definition of "restricted"
:-)

~~~
masklinn
RPython is a restricted (hence R) language specifically for VM development, it
is not a general-purpose language.

------
epanastasi
As someone who works on both python and go day to day, I find this to be quite
interesting.

Just tried this out on a reasonably complex project to see what it outputs.
Looks like it only handles individual files and not any python imports in
those files. So for now you have to manually convert each file in the project
and put them into the correct location within the build/src/grumpy/lib
directory to get your dependencies imported. Unless I missed something
somewhere.. The documentation is a bit sparse.

Overall I think the project has a lot of potential and I'm hoping it continues
to be actively developed to smooth out some of the rough edges.

~~~
ericfrederich
I hope this is a well thought out solution that can evolve into something
great... and not just something built for a single purpose.

I question the transpiler. I think I'd much rather prefer a solution like
Jython.

~~~
Floegipoky
I'm confused because Jython runs on the JVM, but Go is a compiled language.
Can you clarify?

~~~
bb88
Jython is a python interpreter written in Java. Grumpy is a python transpiler
that converts python to navtive go object code.

Edited to add: The difference is that Jython doesn't covert python to JVM
bytecode.

~~~
weberc2
What's the advantage of writing an interpreter? Go already has an excellent
runtime (scheduler, GC, etc)--why should this project reimplement it?

~~~
solidsnack9000
The interpreter could still use that stuff.

One advantage of an interpreter in general is that one important use case for
Python is interactive scripting, as data scientists do.

~~~
weberc2
Fair point. I would think it shouldn't require too much work to build a REPL
on top of this. Rather than transpiling, you would parse the Python AST into
the same runtime Objects that Grumpy constructs statically. Seems
straightforward conceptually, though I'm sure it would be complex in practice.

------
jerf
For those who are interested, I've used grumpy to compile the following Python
code and placed it at
[https://play.golang.org/p/YP1SP7WsdR](https://play.golang.org/p/YP1SP7WsdR) .
(Note the playground can't run this, it just had convenient formatting support
for Go; the generated source wasn't 100% gofmt compliant.)

    
    
        class Test(object):
            def __init__(self, value):
                self.value = value
    
            def method(self):
                print(self.value)
    
        class Test2(Test):
            pass
    
        t = Test("hello")
    
        t.method()
    

Pythonistas, note I had to have "class Test(object):" and not just "class
Test:". The former compiled successfully into a Go program but that program
then failed at runtime with "TypeError: class must have base classes".

~~~
Animats
That's fascinating. It's creating run-time data structures similar to
CPython's for data, and manipulating them with very general code. There seems
to be a type comparable to Python's internal CObject, and it's used for most
(all?) data. It's not generating Go that looks anything like human-written Go.
There's no sign of type inference, although it's hard to tell from such a
simple example. It's a lot like a Python run time environment, where
everything is a CObject. Still, once you can do that, you can start
optimizing, such as inferring that something is an integer and using ordinary
Go arithmetic types.

All that stuff with "switch" seems to be to handle Python exceptions in a
language that doesn't have exceptions. Maybe later, analysis can tell that
some function can't raise an exception, and translated calls for such
functions can be simpler.

~~~
weberc2
> Still, once you can do that, you can start optimizing, such as inferring
> that something is an integer and using ordinary Go arithmetic types.

I was hoping for something more aggressive even, like compiling Python classes
to Go structs so long as the program doesn't need the dynamic behavior.
Alternatively, Grumpy could support declaring native Go types via some sort of
pragma or a new `struct` keyword or some such, which would be treated like a
normal Go object (rather than defining your Go objects in a separate Go
package).

~~~
Animats
I'd expect to see that in time. If you analyze the whole program to find all
the fields of an object and verify the absence of code which dynamically adds
a field, you can then make it a struct of "CObject" like entries. Then, try
type inference on the fields. Some will clearly be integers, booleans, floats,
or strings. Those can be represented with type-specific representations.

If you can identify the built-in types, that's most of the potential win; you
get to do hardware arithmetic. If you represent integers as 64 bits and check
for overflow, you probably don't need bignum promotion outside of crypto code.

------
Pxtl
I can't help but see this balkanization of Python as a sign that the core
language is falling apart.

How many interpreters are there now? And how many of them have even _close_ to
100% compatibility with Python 2.7 or 3.N? Guido has lost control of the
language, but has he's still _officially_ the BDFL there's no real
standardization body. His stubborn view on functional mechanisms have held the
language back syntactically, breaking BC with Python 3 without fixing the
language's fundamental problems... it really feels like Python is lost in the
desert.

Which doesn't mean the language is dead, but it's rudderless. I think we were
all hopeful when Guido joined Google that we'd see real direction for Python,
but that obviously didn't happen.

Not that Python is _dead_ , obviously - still lots of great projects are
written in Python. But I don't like the language's future.

~~~
BuckRogers
GVR, the PSF and the core dev team overestimated their influence. They still
truly believe the majority will come around to Python3. I agree with your
sentiments and balkanization is the right word. Guido won't even read these
comments. He thinks it's all some unjust slander and nonsense that will be
forgotten in 3 years. :)

It is a good time to jump off the Python train in general, and I say that as
someone invested in Python who loves it. If possible I'd recommend people
reach for Go or Elixir depending on their needs or requirements.

I will admit I'm a little shocked how much of a failure Python3 adoption has
been. I think if it had been Grumpy from the start it would've been a huge
success. This is exactly what people want and Google should be commended for
sharing this.

Here's to hoping Grumpy takes on a life of its own and is the new de facto
Python.

~~~
robohamburger
Both my workplace and the one big open source project I use homeassistant all
use python 3. Do you have any data to backup python 3 adoption being a
failure? I know it was certainly painful for many years.

As a python 3 user everything _seems_ fine on my end. Though 3 has its own new
warts. They are smaller and more forgivable warts for now but its probably not
a good sign.

I do agree the direction python is heading is not very interesting anymore but
that doesn't mean its dead or useless now.

~~~
dx034
We try to use Python 3 at work but have to run Python 2 as well because there
are still packages that weren't upgraded and it's too much working re-writing
them all for no direct benefit.

We also have third party vendors that only support Python 3 in experimental
versions, and there not even recent versions (Bloomberg is a great example).

I really like Python3 features, but the pain of using them drives me towards
using other languages. I hope Julia will be stable and mature enough soon so
that I can dump Python all together. I really like Julia, but currently the
changes in the languages are too fast and there are constantly
incompatibilities with packages that don't update fast enough. But I'm
reasonably certain that this will be fixed once they reach 1.0.

I'm sure Python is far from dead, but can imagine that Julia has the potential
to kill it in many domains.

~~~
whyrt12
Do you do data science?

~~~
dx034
Yes, mostly for finance.

~~~
statsmatscats
What python 3 features are painful and why?

------
ericfrederich
Python needs a new runtime. This talk shows how bad of shape it's really in.

[https://www.youtube.com/watch?v=qCGofLIzX6g&list=PLRdS-n5seL...](https://www.youtube.com/watch?v=qCGofLIzX6g&list=PLRdS-n5seLRqszBqVDF342RMlCWgOTm6q&index=11)

Basically, the language doesn't have a "spec" per-se. The language is whatever
the defacto CPython implementation happens to do within it's giant eval loop.

Another great talk about CPython internals:

[http://pgbovine.net/cpython-internals.htm](http://pgbovine.net/cpython-
internals.htm)

~~~
dijit
cpython is the reference implementation, so it makes sense that it's;

A) Not well optimised.

B) Touting features before the spec/standard.

EDIT: people really dislike that I said this, and I'm having trouble finding
my original citation- it was on one of the many python books I own. Most
likely "Learn Python The Hard Way" but I'll dig out the exact chapter where
they compare pypy to cpython and mention that because cpython is the reference
implementation it values code clarity over performance optimisation.

~~~
paulsmith
Who says CPython is not well-optimized?

CPython is 25 years old -- people have been making it faster for a long time.
Python 3.6, the latest release, has many performance improvements, cf.
[http://www.infoworld.com/article/3120952/application-
develop...](http://www.infoworld.com/article/3120952/application-
development/hands-on-python-36s-speed-boost-matters.html)

~~~
trotterdylan
Interestingly, achieving performance parity with CPython is one of the biggest
challenges of this project. There are certain things CPython does very fast
like allocating and freeing many small objects.

~~~
webmaven
So, Grumpy currently _isn 't_ faster than CPython?

------
sfifs
The main reason I moved from coding in Python to Go as my main language many
years back is because concurrency was such a pain in standard Python (the
other was compile time error checking).

It's interesting to see the same pain has now made caused the runtime itself
to be implemented in Go.

It's a pity C extensions (often used in scientific computing) are not
supported but Go does have support via CGO, so maybe some approach can be
worked out to access C routines in the future.

~~~
lelandbatey
I "grew up" on Python, then wrote a whole bunch of Go for my job. Then this
past Autumn re-visited Python to implement a networked terminal based game[1].

With what I learned about Go and concurrency, I would say that currently in
Python, writing _concurrent_ code is not very hard, and is as close to Go as
you can get without actually just writing Go.

Now, you may be saying "but Python has the GIL, how can concurrency be easy in
Python?" I'd say, you're definitely not wrong that the GIL is a problem, but
it's not much of a problem for _concurrency_.

This goes back to the heart of Rob Pike's classic talk, "Concurrency Is Not
Parallelism"[2]. To quote Wikipedia:

    
    
            In computer science, concurrency is the decomposability property of a
            program, algorithm, or problem into order-independent or partially-ordered
            components or units.
    

In Python, you can pretty easily emulate the conceptual properties of
Goroutines and Go channels with Python threads and queues. The problem is that
doing this in Python won't net you the performance increases you get with Go.
And I believe that is an important distinction. There are plenty of cases
where you don't care so much about the performance benefits of parallelism,
but you want the conceptual and implementation benefits of concurrency.

In closing, concurrency in Python is pretty easy to work with, it just
performs very poorly.

[1] -
[https://github.com/lelandbatey/defuse_division](https://github.com/lelandbatey/defuse_division)

[2] - [https://blog.golang.org/concurrency-is-not-
parallelism](https://blog.golang.org/concurrency-is-not-parallelism)

~~~
rcarmo
I wrote a little task module that does precisely that and provides you with a
go() function and "channels":

[https://github.com/rcarmo/python-
utils/blob/master/taskkit.p...](https://github.com/rcarmo/python-
utils/blob/master/taskkit.py)

Obviously, it wasn't amazingly performant. But it did help a lot for doing
concurrent stuff, and I've been pondering re-doing it for asyncio.

------
oblio
It seems interesting but my concern is that it's just another Unladen Swallow.

~~~
the_duke
Well, if Google is pushing it and actually using it internally...

They also have the benefit of being able to push features into the Go core
that they might need for this.

~~~
ProAm
> Well, if Google is pushing it and actually using it internally...

It will be silently abandoned in 18 months....

~~~
ehsankia
Not if it turns out to be usable and have a significant impact on the
performance. You need to realize that at the scale Youtube works, even a small
performance boost translates to huge savings in server cost.

So the worst case here is if they never improve this past their own needs
(which is a pretty limited subset). But if it's successful for them, and it's
opensource, I could very easily see other people who run heavy stuff on Python
contributing to it and helping it grow.

That's the thing with opensource, even if Google doesn't actively work on it,
others can (if it has actual value and is useful to people).

------
zigzigzag
How does this compare to Jython? The blog post says they looked at alternative
runtimes but didn't like that they had tradeoffs (implying that Grumpy has no
tradeoffs??). But Jython has been around for quite a while and also has no
GIL:

[http://www.jython.org/jythonbook/en/1.0/Concurrency.html](http://www.jython.org/jythonbook/en/1.0/Concurrency.html)

It can also handle Python's dynamic aspects.

~~~
Twirrim
Every time I've tried to use Jython, I've found it won't work with pre-
existing code all that well.

In part it has exposed CPython "implementation quirks" that people were
wittingly or otherwise taking advantage of. In other cases there doesn't seem
to be obvious reasons for the differences and has required special-casing the
python code to handle it.

It has been great with code written from scratch, specifically for it.

~~~
zigzigzag
That sounds like the kind of issue you'd have with any reimplementation of
CPython though. It sounds like Grumpy doesn't support quite a few things, so I
still wonder how they compare and why developing a new runtime was considered
easier than reusing Jython.

~~~
toyg
There might be a strategic element: Jython development has been erratic
throughout the years.

~~~
zigzigzag
Google's own projects have been developed erratically throughout the years.
They could have simply forked it.

------
naasking
I wonder why the Grumpy Fibonacci is so much slower than CPython for 1 thread.
Seems weird given Grumpy is compiled.

~~~
machbio
Guido insists that single threaded performance of Python has the highest
priority -
[http://www.artima.com/weblogs/viewpost.jsp?thread=214235](http://www.artima.com/weblogs/viewpost.jsp?thread=214235)

I am not sure, any implementation of Python will beat the single threaded
performance of Cpython..

~~~
Veedrac
CPython is still a naïve interpreter. PyPy beats it in most cases, sometimes
by a lot. Of course, PyPy still has a GIL for the same reasons CPython does.

------
ot
There's no mention of whether Grumpy passes the CPython test suite. Until it
doesn't, it's not a Python runtime, it's a compiler for a language with
Python-like syntax.

Compilers like this, from almost-Python to say C/C++, have existed for a
while: Cython, Shedskin, Nuitka are some examples.

~~~
trotterdylan
It does not, yet. The standard library is very incomplete at this point.

~~~
gigatexal
experimental ~= alpha, give the team a break

~~~
weberc2
You're responding to the principle developer. :)

~~~
gigatexal
I know, but i figure people will read his comments first and then see this.

------
Twirrim
This seems like an odd engineering choice. Presumably the effort to create a
python->go translator would be non-trivial. Why not just start rewriting
components into Go, and migrating them out of Python, leaving python as
essentially the presentation layer at most?

~~~
jerf
Ask yourself how much Python code you think Google has. Then multiply it by
some large single-digit number, minimum. Then compare the effort of spending
30 seconds per line on that code vs. writing a new Python
interpreter/compiler/runtime, especially when you can trivially get people who
are capable of doing that.

There's a reason why the large companies often end up working on new
runtimes/interpreters/compilers like HipHop for PHP, Hack, and so on, rather
than working on the code bases written in those languages. It is very easy for
it to not just be easier to leverage in at that level, but an order of
magnitude or two easier. Or three.

~~~
falcolas
I wouldn't think that a tool for transpiling Python to Go would be harder than
creating a whole new runtime. Most of the work is already done; you can get
the compiled ASTs directly out of Python itself.

Of course, if it started out as a "this seems like a cool project", that skews
the "a is more efficient than b" ratio significantly.

~~~
jerf
Sorry, I think my post may have been excessively specific; I've edited it to
say "writing a new Python interpreter/compiler/runtime" instead of just
"writing a new Python interpreter". I was trying to make a generic point, not
one specifically about a Python _interpreter_. (At least, if I'm interpreting
your reply correctly.)

------
theandrewbailey
This seems like a win for Go, at least to me.

I wrote a random sentence generator in Python several years ago. A bit later,
I wrote my blog using Java EE. Early on, I had an idea: put that generator in
Jython, and spit out a random sentence on every request. It's probably the one
feature that I was OK to let go, should I switch platforms.

Since Oracle has only gotten more evil and Java more stagnant over the years
(especially in light of TLS features), I've been thinking of possible
alternatives. I've been intrigued by newer compiled languages, and it's come
down to either Go or Rust, but I've yet to dig too far into them. I might have
a winner.

------
bootload
_" The downside is less development and deployment flexibility, but it offers
several advantages. For one, it creates optimisation opportunities at compile
time via static program analysis."_

Optimisation. This is a smart move, hard though. A compiler, written well
allows the back end to improve the code. So the whole code base can improve
with improved analysis.

 _" The biggest advantage is that interoperability with Go code becomes very
powerful and straightforward: Grumpy programs can import Go packages just like
Python modules!"_

Extending Python (youtube codebase) with Go modules. That's interesting.

------
thedjinn
I'm curious to why they started this project when there already is low hanging
fruit that can speed up Python (e.g. pypy). What makes this better other than
to satisfy the inner go-fanboy?

~~~
alrs
Because they want to get off of Python completely, and forever. They're not
looking for speed, they're looking to ditch the liability of supporting Python
and its ecosystem.

~~~
JPKab
Yeah, they didn't at all say that.

~~~
randomdata
Of course, that doesn't necessarily mean it is not true. There are many
reasons why Google may not want to reveal long-term plans so explicitly.

It's clear that existing Python codebases will be maintained for the
foreseeable future – there would be no reason to build this otherwise – but
this may signify a shift away from Python for new extensions to the project,
as this now makes it possible to integrate Go packages with relative ease.

------
gamesbrainiac
In case anyone wanted the github repo:
[https://github.com/google/grumpy](https://github.com/google/grumpy)

------
aibottle
This actually pretty great. Even if you will never run your code in grumpy, it
underlines the status of Python. Having large corps investing in Python will
help pulling in new talent into the Python environment also it strengthens the
ecosystem of Python.

~~~
the_duke
Actually, I would interpret this as "CPython is broken for scalability. It's
not worth trying to fork/fix it, we'd rather just roll our own runtime".

Not exactly an endorsement.

Sadly, the code was just dumped into a new Git repo, so no way to tell how
many people contributed internally so far.

~~~
ericfrederich
I agree and disagree ... on one side this may signal a shift away from Python.
Something to help the migration effort and the ultimate goal is to write
everything in Go.

... but then I don't see Python going anywhere anytime soon. Didn't Microsoft
just start a project to get Python's runtime to use CoreCLR's JIT?

There was an article, can't find it now, about an upcoming Python renaissance
saying there may be an influx of new interpreters. There's PyPy, Microsoft's
CoreCLR thing, now this, etc. It seems people really want to program in Python
so there is an effort to make it faster.

*edit: found the article: [https://lwn.net/Articles/691070/](https://lwn.net/Articles/691070/)

~~~
the_duke
Devs love Python and will continue to do so, no question there.

I'm just afraid the surge of different compilers and interpreters will bring
up plenty of issues in the medium term.

There is no formal spec of Python like there is for, eg, Javascript. (which is
of course driven by multiple, VERY engaged adopters).

How long until subtle and not so subtle differences creep in between different
implementations, leading to incompatabilities and a continuous fragmentation
of the ecosystem?

------
scrollaway
That looks like a super interesting runtime. Seems to target 2.7 only, I hope
they're open to supporting 3.x as well.

~~~
trotterdylan
I'd like to support 3.x at some point. See
[https://github.com/google/grumpy/issues/1](https://github.com/google/grumpy/issues/1)

~~~
statsmatscats
Have you thought about using type hints to help type inference?

Some work on that is discussed here. I would love a dropbox google colab
(though also targeting 3.x :) )

[https://github.com/python/mypy/issues/1862](https://github.com/python/mypy/issues/1862)

~~~
trotterdylan
Yes, leveraging type hints for optimization purposes is a long term goal.
Thanks for pointing me to that issue, I'll keep an eye on it.

One of the goals of open sourcing was to get feedback and work with outside
folks so I'm definitely open to collaboration!

~~~
statsmatscats
Awesome!

------
teabee89
From looking in the code, this is my favorite part so far:
[https://github.com/google/grumpy/blob/75a35b4b30afb049b9cfff...](https://github.com/google/grumpy/blob/75a35b4b30afb049b9cfff756bfd6ad6f805a5a7/runtime/core.go#L666)
making python threads as lightweight as goroutines must be a breeze!

------
agentgt
* I would love to know how big the codebase is.

* It seems like writing a translator to deal with all the use cases is so much more work and risky than iteratively rewriting portions (in whatever faster more concurrent language) and using some form microservice/process message passing to communicate with legacy pieces.

* Love to know how they compose async operations currently? Is it some sort of object (e.g. Futures, promises, observables, etc)? Is Grumpy going to have some sort of language difference (to Python) to compose async stuff (e.g. async and await)?

Of course being biased towards the JVM (since I know it so well) they could
get really fast concurrency if they want with Jython today. Most of the Python
tools already work with Jython (assuming 2.7).

With Jython you could always drop down into Java (or any other JVM lang) if
you need more speed as well C for cpython (or even C from Java). It is unclear
what you do with Grumpy with performance critical code. Can you interface with
Go code or is the plan C?

~~~
trotterdylan
> I would love to know how big the codebase is.

Sorry, can't be very specific, but rewriting all the frontend code would take
a lot more effort than writing a new Python runtime :)

> It seems like writing a translator to deal with all the use cases is so much
> more work and risky than iteratively rewriting portions (in whatever faster
> more concurrent language) and using some form microservice/process message
> passing to communicate with legacy pieces.

We do iteratively rewrite components as well. We are pursuing multiple
strategies.

> Love to know how they compose async operations currently? Is it some sort of
> object (e.g. Futures, promises, observables, etc)?

Most async operations are performed out-of-process by other servers.

> Is Grumpy going to have some sort of language difference (to Python) to
> compose async stuff (e.g. async and await)?

I'd love to support async and await at some point.

> Of course being biased towards the JVM (since I know it so well) they could
> get really fast concurrency if they want with Jython today. Most of the
> Python tools already work with Jython (assuming 2.7).

We did also do an evaluation of Jython but there were a number of technical
issues that made it unsuitable for our codebase and workload. One such example
is this longstanding issue:
[http://bugs.jython.org/issue527524](http://bugs.jython.org/issue527524). I
just noticed the very recent update on that thread that implemented the
workaround outlined in 2010 by Jim Baker. We tried that workaround and found
we got a huge performance hit on affected code. There were a few other general
performance problems as well but I can't recall all the details.

Please note I'm not at all bashing Jython, I think it's a great project with a
sound design, it just wasn't right for us.

> With Jython you could always drop down into Java (or any other JVM lang) if
> you need more speed as well C for cpython (or even C from Java). It is
> unclear what you do with Grumpy with performance critical code. Can you
> interface with Go code or is the plan C?

You can interface with Go code directly, e.g. from the blog post:

    
    
      from __go__.net.http import ListenAndServe, RedirectHandler
      handler = RedirectHandler('http://github.com/google/grumpy', 303)
      ListenAndServe('127.0.0.1:8080', handler)

------
ohitsdom
Very interesting project and technical solution. Is this in use at Google? The
described example problem is Youtube with millions of requests per second, but
the post doesn't say if it Grumpy was put in production (and what performance
gains were then achieved).

~~~
trotterdylan
As has been said by others, Grumpy is not used in production at Google
currently. There's still a lot of work to do -- especially on the standard
library -- to support large real world codebases.

------
matthewrudy
I wonder if future work might support C Extensions in the same was as the
jruby truffle / graal implementation plans to do.

Here's a great article from a couple of years ago by Chris Seaton on this
topic.

[http://chrisseaton.com/rubytruffle/cext/](http://chrisseaton.com/rubytruffle/cext/)

------
bigato
I see some parallel with the work made to automatically convert Go compiler C
code to Go code. Strategically wise, I'd rather have the software do the
transpilation to the new language and then after extensive testing, ditch the
old python codebase. But they probably don't agree with my preference of Go
over Python.

------
the_duke
Haha, now that's what I call an interesting project.

The biggest surprise for me is that the Go runtime would be a good fit for
Python, performance wise, considering the very different object and dispatch
model.

The post also mentions runtime reflection, which used to be painfully slow
last time I used it. (Go 1.5, i think).

Has this improved in the latest releases?

~~~
jerf
A huge part of the reason Python (and Perl, PHP, etc. in their original
interpreters) is so slow is that it is like running a Go program that does
_everything_ through runtime reflection, even just to add two integers. If
your code is already in Python, this level of performance is apparently not a
problem for you.

If you know Go or are willing to learn about Go and reflection, you can learn
a lot about how dynamic languages work under the hood by implementing:

    
    
        func Add(interface{}, interface{}) interface{} { ... }
    

using the reflect module to accept all types of numbers, including for a bit
of extra fun the math.Big* number types, and returning upgraded numbers as
appropriate, or panicking on types you can't Add with. That's not all there is
to writing a dynamic language interpreter, but I'd say you can learn the core
idea this way, shorn away from a lot of accidental complexity and with a lot
of the grunt work plumbing of setting up (type, value) pairs already done for
you.

------
jamestimmins
Does anyone know if there is an easy way to get involved in open source
projects like this? The readme mentions adding PRs, but as someone that
doesn't have much experience working with open source projects like this, I
don't even know where to begin. It sounds like an incredible learning
opportunity though.

~~~
steveklabnik
The first stop is usually a CONTRIBUTING.md Here's theirs
[https://github.com/google/grumpy/blob/master/CONTRIBUTING.md](https://github.com/google/grumpy/blob/master/CONTRIBUTING.md)

There's not a ton there. The next place I'd go is open issues:
[https://github.com/google/grumpy/issues](https://github.com/google/grumpy/issues)

It looks like at the moment they're mostly random bug reports from people who
have tried Grumpy since this announcement, rather than ones filed by people
working on Grumpy since before it was made public. So that's a bit trickier.

The last place I'd look is then, the README:
[https://github.com/google/grumpy](https://github.com/google/grumpy) not a ton
there about ways they wish to have people contribute.

At this point, what I'd do personally is open an issue asking how you can get
involved; possibly by improving this documentation on how to get involved!

Anyway, that's what I'd do. Hope that helps!

~~~
jamestimmins
That's super helpful; much appreciated!

~~~
steveklabnik
No worries. Open source can be tricky to get into, but it's largely about just
keeping at it: A story I like to tell is how my first ever PR to Rust was
actually rejected based on a procedural issue. Now I'm on its core team. It
might take you a while, but if you keep at it, I'm sure you'll figure it out.

------
quotemstr
I'm a bit disappointed that the blog post doesn't explain why the authors
didn't choose PyPy instead.

~~~
ot
The objective of the authors is to efficiently exploit multi-core parallelism.
PyPy still has a GIL. They have been doing some experiments with transactional
memory, but performance is quite bad.

------
rpedela
How does Grumpy handle Decimal in Python? As far as I am aware there isn't an
equivalent in Go.

~~~
brettcannon
The decimal module in Python 2.7 is implemented in pure Python:
[https://github.com/python/cpython/blob/2.7/Lib/decimal.py](https://github.com/python/cpython/blob/2.7/Lib/decimal.py)
(in Python 3 there is an accelerated extension module that's used when
available, falling back on the pure Python version otherwise).

~~~
webmaven
So, _does_ Decimal work with Grumpy (you're implying it does)?

For that matter, does Grumpy match CPython's Float behavior exactly?

ie.:

    
    
        >>> 2.2 + 3.1
        5.300000000000001
        >>>

~~~
brettcannon
I'm implying that they can get the decimal module into Grumpy by implementing
enough of Python to support the module (i.e. they don't need to implement the
decimal module from scratch because the module isn't implemented only in C).

~~~
webmaven
Got it. Thanks for the clarification, Brett.

------
0xdeadbeefbabe
> Once we started going down the rabbit hole, Go seemed like an obvious
> choice...

Good? It's funny that a rabbit hole is the place you go to make obvious
choices. This culture must seem strange to outsiders.

------
drej
It's all about performance and concurrency here, but I'm quite happy about
this for another reason: static compilation. If I needed really performant
code, I'd write it in Go/Cython/[insert your fav language] in the first place.
But if I have some existing code that I just want to run without worrying
about the Python runtime being installed/correct, this is a good solution.

(Sure, none of my code will work, because it's all Python3 and usually uses
C/asm, but it's still early and I'm hopefull.)

------
ericfrederich
There was also this effort:

[https://blog.heroku.com/see_python_see_python_go_go_python_g...](https://blog.heroku.com/see_python_see_python_go_go_python_go)

------
sea6ear
Reminds me of the (old I guess?) ShedSkin project to automatically translate
Python to C++.

I suppose the easy concurrency and ability to inter-operate with Go libraries
is really the driver for Go over that.

~~~
Arkanosis
Shed Skin is still living at
[https://shedskin.github.io/](https://shedskin.github.io/) , but hasn't
changed much since the last release in 2013.

------
curtis
One of my complaints about Go is that it seems to be designed with the
(mistaken, in my opinion) notion that what we really need is just a better C.

One way to leverage C's widespread availability and high-performance while
side-stepping some of its deficiencies was simply to use it as a target for a
different language. The Cfront C++ compiler which generated C code is probably
the most famous example, but I recall that there were many others.

Maybe Go, like C, will make a fruitful target for other language
implementations.

~~~
coldtea
> _One of my complaints about Go is that it seems to be designed with the
> (mistaken, in my opinion) notion that what we really need is just a better
> C._

It was designed with the notion that all _some people_ really need is just a
better C.

For other people there's Swift, Rust, etc.

~~~
curtis
For better or worse, I think Go sucked up a lot of the metaphorical oxygen for
similar languages. Swift and Rust are in different niches and don't compete
head to head with Go, so this isn't a real problem for them.

What I've long wanted is a better _C++_ , not a better C. It would be hard for
any such language to succeed today, because it will have to compete with Go,
and Go has a fairly mature toolchain and widespread adoption and a lot of
mindshare.

D literally is designed to be a better C++, and it hasn't really been a big
success. Maybe that's because there really wasn't as big a demand for a better
C++ as there was for a better C. On the other hand it may have been because Go
sucked up a lot of the oxygen that D was going to need to succeed, and maybe
that was because Go was a product of Google and D wasn't. (If this was my
primary point, I'd make a more nuanced argument, though.)

I'm now wondering if the best way to get to a better C++ might be by piggy-
backing on the Go toolchain. To get back to my original point, I'm wondering
if Go might in fact be a good target for all sorts of better (for some
definition) programming languages.

~~~
dochtman
It seems likely to me that Rust is your better C++. I'm also quite positive
that there is (more than) enough oxygen for Rust to succeed.

As for D, I think it has numerous other issues, that don't apply to Rust.

~~~
curtis
The stuff I've done in C++ would have worked as well or better in a language
with garbage collection. On the other hand if you want to write the next OS
kernel, then you need a "better C++" that is designed to work without GC. In
the latter case, I think there is no question that Rust is your best bet
today. I'd like a programming language that is statically compiled, with good
performance, garbage collection, and decent support for abstraction. It's the
last part where I think Go comes up short, and I think the Go designers think
that's a feature not a defect.

~~~
dochtman
> I'd like a programming language that is statically compiled, with good
> performance, garbage collection, and decent support for abstraction.

It feels like you're mixing actual requirements (like "good performance" and
"decent support for abstraction") here with things that are more like
implementation details ("garbage collection", "statically compiled"). I don't
really understand why you want "garbage collection", as such, though I could
understand wanting some kind of increased productivity from not having to
think about memory management. However, if your real requirement is as I
describe it here, then I would still contend that Rust fits the bill.

------
walrus1066
Question: better threading performance seems to be the main motivation for
this project. Could they not just use multiprocessing instead of threading for
the cpu-heavy parts of the YouTube codebase, and threading or asyncio for the
io-bound parts of it?

Edit: for example, the fib benchmark they cite is cpu-bound. If the python
code used multiprocessing, the performance would scale almost linearly with
the number of processes.

~~~
webmaven
_> Could they not just use multiprocessing instead of threading for the cpu-
heavy parts of the YouTube codebase, and threading or asyncio for the io-bound
parts of it?_

Probably, but consider how much bare iron Google has lying around, much of it
likely with few cores-per-CPU. Using it efficiently and not accelerating it's
obsolescence is probably a priority for them.

BTW, does anyone have data that would suggest how long it does take an org at
the scale of an Amazon, Google, or Facebook to entirely replace their HW? I
assume that it isn't only through attrition, and that Google for example
currently has no servers running that date back to Y2k, but I have no idea
what the "half-life" of a server is at their scale.

------
agounaris
Whatever the haters say here, this is really cool!

------
zedpm
I would love to see this support Python 3.5+, specifically for asyncio.
Concurrency via threads in Python is far less appealing.

------
webmaven
The code initially landed on Github 16 days ago[0]. How long was it developed
internally before that?

[0]
[https://github.com/google/grumpy/commit/f60ee257db9d7996e3f8...](https://github.com/google/grumpy/commit/f60ee257db9d7996e3f8da148260e113516e3a90)

------
cbhl
The blog post says that YouTube is the inspiration for this runtime, but does
YouTube run on Grumpy yet?

~~~
trotterdylan
YouTube does not run on Grumpy. There is a lot of work left to do before
Grumpy can run a large existing codebase.

------
weberc2
I tried a simple `http.Get()` example, but the resultant `Response` object
appears to not allow access to any of the struct fields as attributes. How
does one access a Go object's fields from Python? For example, I want to
`io.Copy(os.Stdout, rsp.Body)`.

~~~
trotterdylan
Unfortunately the native interface is still pretty immature so I can't
guarantee things will work as they should. In this case, I think the problem
is that you have a *Response object which is not itself a struct. I've
rewritten this part of the code a couple times and I think this functionality
was lost. I've filed:
[https://github.com/google/grumpy/issues/13](https://github.com/google/grumpy/issues/13)

~~~
trotterdylan
Whoops, just noticed you (or somebody) had already filed it:
[https://github.com/google/grumpy/issues/12](https://github.com/google/grumpy/issues/12)

~~~
weberc2
Yeah, that was me. I should have updated my post here accordingly. Apologies
for the inconvenience!

~~~
trotterdylan
It's all good. Thanks for filing the issue. I'll get that fixed.

------
nodivbyzero
Facebook: PHP -> HipHop

Google: Python -> Go

------
kristianp
Insightful discussion in the comments here about performance and JITs
[https://lwn.net/Articles/710634/](https://lwn.net/Articles/710634/)

------
waitingkuo
Will Grumpy be a good fit for go's scientific packages such like gonum?

~~~
trotterdylan
I don't know. I hope so. I think this is an area where Go can succeed and
integration with existing Python libraries could be useful.

~~~
waitingkuo
This is amazing work. Look forward to the integration between go & python

------
stiff
It's actually a transpiler written in Python that generates Go code.

~~~
kodablah
Nice. I've been toying with doing similar for the JVM [0]. I think people are
coming to realize that while Go is not a great/powerful language (IMO), the
runtime is great. I would not be surprised if more and more people start
targetting Go if they want a GC and cross platform static compilation w/out
LLVM complications and get a nice stdlib for free.

0 - [https://github.com/cretz/goahead](https://github.com/cretz/goahead)

~~~
bsaul
First to write a language that is basically go + generics wins

------
statsmatscats
Will you continue maintenance on this indefinitely (barring orders from higher
up) or is this meant for a one shot translate and dump?

------
gigatexal
on a related note: given my affection for python and having, until just now,
no idea that YouTube runs python (albeit 2.7) at Google scale (a language not
known for being able to scale very well given the GIL and such)-- I now more
than ever would love to work on something like that. Welp, back to learning
and hacking with python.

------
tiffanyh
>> "The front-end server that drives youtube.com and YouTube’s APIs is
primarily written in Python, and it serves millions of requests per second!
YouTube’s front-end runs on CPython 2.7, so we’ve put a ton of work into
improving the runtime and adapting our application to work optimally within
it."

So has YouTube already migrated over to using Grumpy (and no longer running
python in production)?

------
Sir_Cmpwn
God, will Python 2 just die already?

------
acd
May I suggest GoPy as name

~~~
vorg
Maybe you don't like Grumpy as a name. I think it was named after NumPy (pr.
ˈnʌmpaɪ) so I guess it's pronounced ˈgrʌmpaɪ rather than ˈgrʌmpi:

Around 2011 the beta releases of the static typing additions to Apache Groovy
were called "grumpy" but the name was dropped after objections from the Grails
crowd. I think the quality of the product is more important than the name, so
Grumpy should do OK regardless if it's built and maintained properly.

------
philip1209
With click.py, this could make amazing command line utilities

------
jsmith_dev
Win for crypto library inclusion
[https://twitter.com/jsmith_dev/status/816894517844418561](https://twitter.com/jsmith_dev/status/816894517844418561)

------
sandGorgon
any chance you guys have tested this with numba?

that would be awesome.

------
sheerun
I'd love something like this for JavaScript

------
BuckRogers
This is the best news to come out of the Python space in a very long time.
Python on the Go runtime, no GIL, more performance, is exactly what people
wanted out of a new Python.

------
codebungl
+1 to cooperative multitasking

------
brian_herman
No C extensions boo.

~~~
MBCook
But isn't that really what lets them do this? In following PyPy (and other
alternate Python discussions) it seems like eliminating the GIL and getting
better performance out of Python, even in C, isn't that hard _if you drop the
C extensions_.

As soon as something can see into the guts of the interpreter you have to
maintain compatibility which is a pain/waste.

Worse than that is that view wasn't designed for multi-threading which is why
the GIL exists. The C extensions were t designed to be multi-threaded because
that wasn't a thing in Python so they're not safe. You either have to drop
them, define a new interface layer that would be safe, or I suppose somehow
sandbox their little view of the world but keep it coherent between threads.

If you have a codebase where you can make the choice to drop C extensions and
you're trying to accelerate Python it seems like a very smart choice.

~~~
guitarbill
The irony is that some C extensions exist because they have better performance
than Python.

On the other hand, there are many C extensions which are just interop/wrappers
for existing C code. I think no language is naive enough to think they can get
away without C interop.

C extensions are actually pretty nice because you can wrap the C code into
idiomatic Python. With ctypes, you have to e.g. maintain two structure
definitions, which is very problematic for some codebases.

> In following PyPy (and other alternate Python discussions) it seems like
> eliminating the GIL and getting better performance out of Python, even in C,
> isn't that hard if you drop the C extensions.

AFAIK, many projects initially struggle to even achieve performance parity
with CPython. The GIL isn't evil, it's a simple solution to a hard problem.
But at this point, a complex solution to a hard problem is better if it's
faster, and some people care more about speed than C interop (and vice versa).

What would be awesome is a Python runtime that could run with fine-grain
locking until a C extension is loaded, and then continue with coarse locking.
But the problem is still there's no upgrade path for C interop.

The only thing I can think of is using type annotations and something like
Cython's cdef to write Python-implementation-independent C interop that
doesn't suck as bad as ctypes. Then the Python rumtime could also lock the
arguments at a very fine level while the function is being called.

------
senthilnayagam
wish there was one for ruby as well

------
MrBra
Sometimes I really wonder what Google has against Ruby.

------
vegabook
2.7 haha. First Tensorflow, now Grumpy. I can't stop laughing as all the 3.x
zealots try to squirm out of this. Talk about awkward.

yes yes I know tf now "does" 3 but we all know what Google really cares about.

"I'd like to support 3.x _at some point_ " \- trotterdylan.

Read: nice to have but when it gets down to brass tacks, 2.7 is where it's at
for Google.

------
Antwan
"So we asked ourselves a crazy question: What if we were to implement an
alternative ?"

As always with Google...

What if you humbly contribute to open source projects rather than creating new
stuffs, labelled with your own brand, controlled by your own engineers, and
with your own design choices, however good they may be ?

~~~
tedmielczarek
They tried that with CPython, it didn't work:
[https://www.python.org/dev/peps/pep-3146/](https://www.python.org/dev/peps/pep-3146/)

~~~
teddyh
But it was no fault of Python: [https://qinsb.blogspot.nl/2011/03/unladen-
swallow-retrospect...](https://qinsb.blogspot.nl/2011/03/unladen-swallow-
retrospective.html)

~~~
tedmielczarek
Interesting, I hadn't seen that post! I wonder if there are any posts that
elaborate on the internal Google bits more. I'd love to know what "they found
other ways to solve their performance problems" wound up meaning in practice.

~~~
webmaven
_> I'd love to know what "they found other ways to solve their performance
problems" wound up meaning in practice._

Me too, especially as this was already several years after the YouTube
acquisition.

Also interesting to note that potential internal customers were put off by
having to upgrade to 2.6.1 in order to use unladen-swallow (presumably they
got over that reticence at some point, as 2.7 is now standard).

------
BuckRogers
Is this the final nail in the coffin for Python3? Seems like it.

Who would use 3 if you could have CPython2 for existing code and write new
code in Grumpy Python? This is the dream language for me. Python on the Go
runtime.

~~~
int_19h
Everyone who likes the numerous new language features, or new syntax for old
features, in Python 3?

~~~
BuckRogers
Those can be built into Grumpy if anyone cares to do so. You don't need
Python3 for that stuff. Someone, one developer, recently released a Python 2.8
that backported almost every new Python3 feature.

I'm hoping it becomes a permanent fork of Python2 that uses the Go runtime.
That would really be great and exactly what they've got now.

~~~
int_19h
If you backport every new Python3 feature to Python2, it becomes Python3, by
definition.

~~~
dismantlethesun
It could become Python 2.8, wherein the standard library has both the legacy
versions and the new versions, and any cross-compatible syntax is allowed.

This leaves us with more than 1 way to do things, like meta-class declaration.

There's a lot in Python 3 where changes were made to the syntax for 'clarity',
but those worts weren't removed for any _technical_ reason but because of the
thought that since backwards compatability was being broken anyways, then we
might as well get the most bang for our buck.

~~~
int_19h
> It could become Python 2.8, wherein the standard library has both the legacy
> versions and the new versions, and any cross-compatible syntax is allowed.

So two `builtin` modules, then?

What if someone does `sys.modules['builtin']`? Or any other kind of explicit
string-based lookup?

How does pickle figure out which types to instantiate? There'd be a lot of
types with same qualified names but different implementations with this
approach...

It feels like the only way this would work reliably, is if you completely
isolate the Py2 and Py3 universes. So if you e.g. pickle from Py2 code, it
only looks at modules and types that Py2 universe knows, and vice versa.

But then what happens when code using the old library interacts with the new
one (e.g. tries to pass objects around)? If that is prohibited, then you
effectively still have two different languages, just with a single shared
implementation - but no ability to gradually replace bits and pieces of code,
for example, which would seem to be the biggest motivation for such a thing.

