
From Python to Go: migrating our entire API - ainar-g
http://blog.repustate.com/migrating-entire-api-go-python/
======
dv_says
Formerly using Python, motivated to migrate to a modern stack, and so recently
attempted to write a simple web back-end for my apps in Go and then in Elixir.
Same logic, same unit tests, two languages.

Really liked: the simplicity of Go's build, not having to spawn N processes on
the server side, the compiler catching so many issues, the very strict
formatting rules, all a major win over Python. However, didn't like how
verbose the code turned out, especially for parsing or any data manipulation
tasks. For server-side debugging/tools, it's also a bit of a build-your-own
adventure.

Elixir I found to be much more pleasant to write and subsequently read, more
well-suited for back-end tasks, great for generating and parsing data. Also,
more things in the toolset for back-end deployment, in large part due to the
OTP stack.

Perhaps Go's ideal use case is local system tools, compilers, or anything else
that requires concurrency (let's say, creating tools like "ls," or Apache
bench, or gcc, or cURL), and something like Elixir is more ideal for building
actual servers.

~~~
gtaylor
It sounded like they were gunning to shave the maximum in response time from
their HTTP APIs. From my understanding, Elixir/Erlang probably wouldn't be
able to match an optimized Go process in raw efficiency. I get the sense that
scaling horizontally is the big draw of the Erlang VM, not so much raw
responsiveness.

If simplicity were the primary concern, they could have stuck with Python. You
do get a much more distributed element with Elixir/Erlang (which certainly
helps with volume), but I'm not sure they'd reach that 10ms response time
average with it.

~~~
BuckRogers
If latency were the primary concern nothing would beat the BEAM platform. It
was built for that purpose, low latency is absolutely critical when you make a
phone call.

Of course that's a bit like a benchmark (meaning, it means absolutely nothing
to you/your code). If there were any more latency in the Erlang solution it
would be the same algorithmic issues that could pop up in any software.

Go is more efficient for CPU throughput, but BEAM has been fine tuned for
decades to keep latency down and wouldn't be able to top for responsiveness.

My personal opinion is that they made a mistake. Reporting about swapping out
a stack like this is weird because it looks like they don't know what they're
doing. What's next month's blog? We're Moving to Rust? I'm glad they're
satisfied, but you gain improvements anytime you do a rewrite- even in the
same language. I wouldn't have migrated my existing stack anywhere. I'd use
PyPy which would've done wonders for responsiveness on the existing product
and adopt containers for deployment.

~~~
d23
> I'd use PyPy which would've done wonders for responsiveness on the existing
> product and adopt containers for deployment.

That's an easy claim to make, but you don't know whether their stack would
actually get much improvement out of PyPy. We've had trouble in the past using
PyPy with C extensions. Trying to migrate to pure Python equivalents was a
headache. Once we managed to get a PoC of a service running under PyPy, we saw
very little improvement.

~~~
BuckRogers
It is an easy claim to make but just as easy for them to test as well, so it
works out.

------
BuckRogers
I see a lot of these types of blogs but I'm always left with the same
questions. Did they not try using PyPy? Using LXC and PyPy would be my first
step, not swapping out my stack. It just seems more sensible to me considering
that Go as a platform isn't perfect either. Nothing is.

But I'd be hesitant to leave a massive, thriving ecosystem like Python
(especially if on 2.7 which has everything). Win some things, lose some others
when you can just adopt containers and PyPy and gain the benefits that Go has
without losing Python's advantages.

~~~
eloff
They tried Cython first, so PyPy wouldn't have been faster than that. The big
speedup for them was in making the code parallel, which is tricky in python
and typically requires processes instead of threads and copying instead of
sharing, either of which may eat most of the gains in this case.

------
cowsandmilk
They cite their main reason for migrating as being self-hosting, which is hard
in Python.

I've had a huge amount of success using pyinstaller for this problem, shipped
many products into hugely diverse environments.

[https://pyinstaller.org](https://pyinstaller.org)

~~~
melling
I think performance ended up being the big win:

"We migrated our entire API stack from Python (First Django then Falcon) to
Go, reducing the mean response time of an API call from 100ms to 10ms We
reduced the number of EC2 instances required by 85%"

~~~
medecau
I'm usually left wondering if anyone else know about the multiprocessing
module. Specifically the Pool class.

~~~
melling
Go is going to be a lot faster than Python even without multiprocessor. The
performance difference becomes even larger on multi-colored cpu's. Say, for
example, if Go is 3 times faster than Python on one core then you throw in 4
or 8 cores, the win is even bigger.

~~~
BuckRogers
That's true but how many tasks legit need multithreading? Not many in my
experience. Over the course of my career I've come to the conclusion that
multithreading is best left for systems programming and very few specific
applications (such as a game engine and not much else).

For pretty much everything else that people use multithreaded applications for
in the past (encoding, image editing)- multinode processing is vastly superior
to multithreaded. Erlang calling out to C (if even necessary) is going to win
there over most. Or you could use Python or whatever you wanted.

The Java/Go way of local multithreaded applications seems archaic to me.

------
cdnsteve
How did debugging change, for the better or worse? Did your tests speed up?

------
bliti
I wonder why they didn't go with Java. It would have been a good fit with the
additional benefit of not having to pay the premium Go devs go for (Go dev
here).

~~~
danieldk
Did someone gather salary statistics? I could imagine it could also be the
other way around: hordes of Go fans competing for a relatively small amount of
jobs drives prices down.

~~~
eatonphil
Am I the only one who thinks it's crazy that people would compete for jobs
because of a language? Especially a new language? Would anyone seriously give
up a job just for the opportunity to program in another language?

Perhaps this is a bigger thing among corporate programmers? I've only worked
at smaller companies where they are more open to experimenting if you talk
them into it.

~~~
zurn
Many programmers have fairly strong preferences about programming languages
and definitely many people would consider a Go job more desirable than a Java
job. Most of those people aren't currently Java programmers - it's said[1]
that Go is largely attracting people from dynamic languages and not from
C++/Java.

[1] See this essay from Rob Pike where he says Go is attracting programmers
mostly from dynamic languages (though he raises C++ programmers as the chief
contrast, but still mentions Java too) :
[http://commandcenter.blogspot.fi/2012/06/less-is-
exponential...](http://commandcenter.blogspot.fi/2012/06/less-is-
exponentially-more.html)

------
mikesf888
my favorite quote: "Writing software in languages like Python or Ruby often
seduces you into being ignorant of what’s going on under the hood because it’s
just so easy to do pretty complex things, but languages like Go and C don’t
hide that."

~~~
erokar
Damning with faint praise. Often, you don't need to know what's going on under
the hood, and often it's not very interesting.

~~~
marcosdumay
To tell the truth, Python is an easy target here, but I doubt 99.9% of the
projects in C or C++ care to manage memory as closely as GHC does under the
hood. (The same way as they don't manage concurrency.)

Some times, it's even better not to look.

------
Scarbutt
tl;dr - Python is to slow for us. (its about the implementation, not the
language).

~~~
simeonf
I expect the performance/parallelism complaints. It's interesting that the
tl;dr also includes "packaging and distributing an app is a reason go >
python".

Python needs a .war format and I'd like to see more people get behind
[https://github.com/pantsbuild/pex](https://github.com/pantsbuild/pex)

~~~
WorldMaker
The original Python .war equivalent was the .egg, but that's slowly losing
ground to the .wheel.

~~~
WorldMaker
Oh wait, I'm confusing .jar and .war, aren't I? I think .wheel has some .war-
like characteristics, thus it's growing popularity, but an not sure.

------
vkat
Pretty neat trick porting over python unit tests to go. Any plan on sharing
more details or open sourcing that part?

------
Randgalt
"Due to the similarity between Python and Go" \- huh? Go and Python are almost
perfectly orthogonal.

