
Go + Services = One Goliath Project - yawn
https://engineering.khanacademy.org/posts/goliath.htm
======
purple-again
We turned our monolith into a bunch of micro services almost 6 years ago to
the day. For a long time I was very happy with the new pattern but over the
years the weight of keeping everything updated along with the inevitable
corners that fall behind and have...questionable..security due to how long
they sit neglected has really left me wondering if I am happy with it after
all.

I would love hear some thoughts from others that made the move, especially
anyone that decided to move back to a monolith repo.

~~~
scottrogowski
A company I am affiliated with made a decision to rewrite their code in
microservices-oriented architecture thinking it would only take one year. Now
we're 7 years into the transition and starting to come up against some hard
deadlines that threaten revenue streams. It seems obvious to everyone except
the leadership and the architects that this has been an unmitigated disaster.
Other comments on this thread seem to indicate that many have had similar
experiences.

For those who are curious, here is a classic article on why rewriting code
from scratch is a bad idea: [https://www.joelonsoftware.com/2000/04/06/things-
you-should-...](https://www.joelonsoftware.com/2000/04/06/things-you-should-
never-do-part-i/).

For a more in-depth analysis on the unforeseen challenges of microservices in
particular, I would encourage a lot of careful research into how other
companies have tried and failed at this. In particular, I might look at Uber's
ongoing difficulties.

All I have to say to the Khan Academy engineers is to buckle up because
frankly, moving from Python 2->3 is not that hard and you have no idea what
you are getting yourself into.

~~~
andy_ppp
Yes, the micro services + Golang vanity project because you think you’re
Google. I really don’t think people understand error states in distributed
systems very well, putting possible network partitions everywhere is not a
great idea. I would strongly suggest trying a Golang monolith first and seeing
if there are one or two heavily used services that need splitting off. Also
monorepo. Always.

~~~
StavrosK
Microservices have always seemed dangerous to me. "Bugs thrive in the seams
between libraries, so let's put more and deeper seams in!"

Microservices have an immense cost, and you have to make sure they're worth
it. Many teams years ago found it a nice pattern and implemented it because
why not, and now we're at the "oops this isn't actually amazing" part of the
cycle.

~~~
andy_ppp
If you work at Google where the standard is higher fine, but 99% of places
people end up making very buggy stuff that interrelates in weird ways!

------
kureikain
Go is a very weird language :-(.

It's very limited when you started to do complex thing. Example, let's say you
are building websocket. You will have a hard time to write type safe websocket
handler to process the payload from client for all the events...

I started to do Rust/Crystal and both of them are better than Go(performance,
type system).

Yet, whenever I build something for work, I come back to Go :-(. I told myself
to use Rust or Crystal.

Then I realized that Go is a practical language. It compiled fast so it makes
testing easier. The cross compiler just make it so easy to build binary run on
everything thing. And the limitation of Go makes it very consistent on how you
do thing. This makes working with Go become faster event by the fact that it
slows you down on other parts.

So I think Go is a language that people easier to fall into because it has the
speed of interpreter language like Ruby/Python(or even faster) during
development and have a better performance/type safe story.

~~~
apta
Java and C# provide the benefits you mentioned, while being more expressive
languages and having better runtimes compared to golang.

~~~
hu3
Hardly. OP mentioned fast compilation and limited ways to code the same
solution.

C# and Java are slower to compile and offer way more options to do the same
thing.

Here's Uncle Bob take on testing with Go: [https://m.youtube.com/watch?v=2dKZ-
dWaCiU&t=36m40s](https://m.youtube.com/watch?v=2dKZ-dWaCiU&t=36m40s)

~~~
apta
> C# and Java are slower to compile

Not for any meaningful work in my experience. As a matter of fact, I found the
change/compile/run loop in golang to be slower on projects I've been working
on due to the fact that it doesn't support incremental compilation, so any
change I make ends up recompiling the entire program and writing out a 100+MB
binary anyway. Compared to a Scala project I worked on before (and Scala is
notorious for slow compiles), after the first compilation, all modifications
happen very quickly as only the respective classes are re-compiled.

> Here's Uncle Bob take on testing with Go

Again, this doesn't apply for any non-trivial/large project. On a project I'm
working on, it literally takes 7-8 minutes to do a clean build + run all unit
tests in golang.

~~~
hu3
Go absolutely does incremental builds by default and has been like that since
I can remember. Packages are only rebuilt when their source or their
dependencies change.

Same for tests which are cached by default so during typical development only
a subset of tests are executed and compilation time can be a big part. Leave
full tests for CI.

An anecdote I found from 5 years ago:

On my 1.7GHz processor it takes 10 seconds to build the whole standard library
from scratch (300k lines of code).

It's fast AF.

------
plinkplonk
just trying to understand - you guys think moving a Python2 monolith to Python
3 is too painful, and so you are going to port all the code from Python2 to a
completely new language (Go), change the architecture (monolith ->
microservices) and move the HTTP API to React + GraphQL, all in one year?

2020 is going to be in an interesting year at Khan Academy ;-)

~~~
CGamesPlay
The move from Python 2 to 3 would likely have also involved changing the
architecture so they could migrate components incrementally. Since they were
going to do that regardless, and if they already wanted to change the
interfaces from HTTP to GraphQL, this is a natural time to do it. Though, this
migration has nothing to do with React--they were already and will continue to
be using it.

------
bhntr3
> If we moved from Python to a language that is an order of magnitude faster,
> we can both improve how responsive our site is _and_ decrease our server
> costs dramatically.

I see people say things like this a lot but my experience is that while other
languages are 10x or more faster than python in some benchmarks it's very rare
that computation time dominates server latency or that servers are running at
60%+ cpu across all cores.

If 90% of your service latency is not directly on the cpu and/or you haven't
profiled to see that the performance bottleneck is evenly distributed across
all tasks, then it's super dangerous to migrate to a new language thinking
that will fix it.

I hope people inside Khan Academy know this and it's just a clickbait blog. If
they really think "go is 10x faster than python so we'll only need 1 server
for every 10 when we migrate" then I think they'll be disappointed.

~~~
pm90
* It’s not just that Go is faster to run but also faster to iterate on. If python can be neither, its offering little benefit.

* they moved from a monolith to a microservices architecture; concern that any of the services in the request path could add latency just because of the overall runtime speed is slow is a legitimate one.

* their primary deployment method is Google App Engine where you are billed by CPU used. Any change that consumed less CPUs has a tangible effect on their costs

~~~
andy_ppp
Go is faster to iterate on is absolutely false. Where do you get this idea
from?

~~~
pm90
I can easily ask the vice versa of such a bland question. Answer that first if
you want a real answer instead of trying to flame.

~~~
andy_ppp
It’s not bland it’s direct. What you’re doing is conflating personal
preference with actual language features. Most developers who aren’t us would
say python, being a higher level, dynamic scripting language rather than
Golang which is lower level and extremely explicit about the data your program
is using. Iteration in go is simply harder as you have to be more explicit
rather than sketching something out. Changing that more explicit stuff is
harder if you got it wrong while iterating. Why do you think Golang is better
at iteration?

Add in extremely poor error messages, lack of generics, having to generate
loads of code for various things, the ability to crash whole services if your
program does something incorrect, excruciating error handling will all slow
you down.

~~~
pm90
It’s funny that none of the things you mention in your last paragraph are
handled any better in Python. Without type safety, you instead have to deal
with bugs caused by inane mistakes. If we’re talking about generics, the
conversation has shifted from “writing scripts that does x” to “maintaining a
production system” and for the latter, golangs type safety, out of the box
excellent default tooling and easily grokkable concurrency primitives make it
far more maintainable than Python.

Things are also easier to change in go because of the type system and
interfaces. The former catches most the obvious incompatibilities, the latter
ensures that abstractions don’t leak across different system boundaries;
whereas in Python there is a tendency to pass a do everything objects across
the system.

Error checking has improved substantially with error wrapping in go 1.13. Not
only can you locate precisely where your system failed; you have to be
explicit about handling errors. I do concede that pre error wrapping the error
handling was garbage.

~~~
andy_ppp
Clearly I don’t agree but it’s an interesting and detailed answer. I do agree
that type systems help with refactoring but not iteration. There’s a fine line
there. Personally I think Golang’s type system isn’t as good as it could have
been... I do like the idea that Golang reports to provide quite good
(simplicity above everything), if they added proper macros to replace code
generation and to replace the desire I have for generics it would be a much
more useful language for my needs.

------
harel
Migration from python 2 to 3 is easy and fast. I've migrated multiple large
apps and it took about a day each. Most libraries that matter have been
migrated. Some don't even support python 2 anymore. It's practically 2020.
This should not even be a consideration. After 2 to 3 is done they should
consider again If they want to redo the stack but first I'd focus on this
small maintenance task.

~~~
AnonymousPlanet
Hahaha, sorry but this is a very cute thing to say, in my view. At our company
we just barely finished migrating our software with nearly a million lines of
legacy Python 2 code to Python 3. This took over a year of nearly exclusive
migration effort, just making our code work with both. The entire migration
project started way before I joined the company several years ago.

So, no, things are not as simple if you're not dealing with toy projects. And
no, you can't assume that it's the same for everyone if you're not in their
shoes.

Your comment is pretty much the equivalent of "I don't see a bug. Works for
me."

~~~
StavrosK
It very much depends on how good the codebase is. I also spent a year on and
off porting a large codebase from 2 to 3, and it would have gone an order of
magnitude faster if the codebase were in better shape.

~~~
AnonymousPlanet
I agree that code quality is a big factor. But what is good code quality in
Python? In our case, the oldest code is the most "pythonic" and is at the same
time the worst to maintain. The better code mitigates the drawbacks of dynamic
typing and by that moves away from the pythonic standard you see in many
libraries.

But even if you nail the types to the board (e.g. assert isinstance(...)), use
(the somewhat weak) Mypy wherever you can, and have good test coverage, you
still have to grep your code base for usage of, e.g., .keys(), eyeball
hundreds of modules for subtle Unicode madness or hunt for the odd division,
replace every sort() that doesn't use key= yet, etc. The todos add up and
someone has to go into the code and change those lines.

~~~
stult
What specific “pythonic” habits have you found are more difficult to maintain?
Asking out of genuine curiosity, not to challenge the premise. I work with a
lot of data science people that really emphasize being pythonic, but coming
from the software/static typing side of the house I always find their code
style and architecture a little concerning, and I’m not sure if I’m just not
getting it or if they really are writing spaghetti.

~~~
AnonymousPlanet
One of the biggest footguns is method naming. Most Python libraries will
gladly use generic method names like "add()" or "getName()". The moment you
need to rename the method or change the signature, you will have a hard time
telling it apart from all the other method calls by the same name. No type
inference will save you here because type inference is incomplete and will
never let you find all the callers.

What you _should_ do is use unique names. But that will give you ugly code
like myFoobar.foobar_addBar(). The kind of code that makes the pythonic crowd
cringe.

Another problem is making code too generic with regards to what types it
consumes, instead of nailing it down to the few types you're ever going to use
here. This makes it hard to reason about your code months and years down the
line. How is this method used in the rest of the code base? Do all callers
expect an int? What if my method now happens to return a float?

And there's also abuse of duck typing. Throw around a lot of objects,
sprinkling methods and other members on to them as you go. Then when you
consume the object, just look if it has the method you want to call. This
makes any kind of type checking and static type inference useless.

And then there's a whole lot of Python 2 libraries where you get the feeling
that the authors didn't give too much thought about whether they are dealing
with str or unicode. The method might just call .encode(...) on one of its
arguments without being too sure what it is.

And every one of the mistakes that result from the above practices might only
pop up when your code has already been shipped to the customer site.

------
RandyRanderson
Why do they call them "micro services" and not distributed systems? Oh right,
it's because distributed systems are obviously really hard to create correctly
and no sane person would ever agree to pay for that.

Nice: re-branding. I can't wait for the, maybe "consolidated computing"
manifesto (aka turning micro services back into monoliths).

~~~
thethimble
What if the services you are writing are independent in that they solve
separate business problems, are built by separate teams, have little to no
data coupling (e.g. Only basic auth), have different scalability profiles,
etc? Separate services are really effective for these cases. Neither micro
services nor monoliths are silver bullets. Instead it's possible for each
approach to be the best approach in a particular business context.

~~~
RandyRanderson
In several decades of it experience , I've not known or heard of a nontrivial
system like the one you describe in the first part of your message.

In the latter part, that must be a disingenuous dichotomy. You don't really
believe that just because an avenue exists we should include it in an
evaluation?

------
zmmmmm
I think some of the misunderstanding in these comments comes from not fully
appreciating the perspective of not-for-profit organisations. While I can't
speak for Khan Academy, I know that in every NFP organisation I have worked
for there is an acute awareness that funding could dry up one day and the
prime directive is to ensure that in a scenario like that, the work of the
organisation can continue.

In this case, it leads to a higher concern about minimising the cost of the
operational services than you might have in a for-profit organisation. In all
the strategic planning I have been involved in with NFP, we always have the
"what if worst case scenario arises" plan and in that plan the ability to
scale down to bare minimum operational cost is key. It may not be conscious
but I suspect that may be part of the reason the performance savings from
moving to Go are so attractive in this case, where most profit-making
companies just ask the question of whether they can afford to pay for the
servers with their current margin or not and if they can they have more
important things to worry about.

------
apta
> Go, however, used a lot less memory, which means that it can scale down to
> smaller instances.

I could be mistaken, but this sounds like they went ahead with the default JVM
settings, where it tends to use as much memory it is allowed to (which makes
sense from a utilization and efficiency perspective). If memory usage is a
concern, the JVM can be tuned for such.

~~~
jacques_chester
The JVM hasn't yet become fully container-friendly because it bases its
calculation on the host OS figures, not the container figures.

You can use a calculator to get precise, proven settings for any supported
JVM: [https://github.com/cloudfoundry/java-buildpack-memory-
calcul...](https://github.com/cloudfoundry/java-buildpack-memory-calculator/)

~~~
apta
I thought this was a solved issue since JDK 10:
[https://www.docker.com/blog/improved-docker-container-
integr...](https://www.docker.com/blog/improved-docker-container-integration-
with-java-10/)

~~~
jacques_chester
In my understanding, no. It's improved but not fully "fixed". Unfortunately
that's as far as my understanding extends.

------
peterwwillis
So, some potential pitfalls:

\- The decision seems to be primarily a software architecture one, without
much mention of all the other architects whose input will shape how the
finished product is run and supported. In a modern software development
environment, all the other parts of the org should be consulted on greenfield
work to "Shift Left" anything that may need to change down the pike. Design in
a silo leads to ineffective products.

\- They're going from "hmm we need to upgrade from Python 2 to Python 3", to
"we need to redesign everything in a new language with a radically different
software architecture". This is definitely the second system effect. It's
going to take years to make this thing reliable and sunset the old product.

\- They're porting over the logic? Even if this is actually the right move,
wouldn't a clean-room implementation potentially give better outcomes?

\- Why are they continuing to use App Engine if the writing's on the wall for
2024?

~~~
dangoor
I don't disagree with your pitfalls, but I do think we're working to avoid
them.

To your first point, "The decision seems to be primarily a software
architecture one...", this project has had involvement of the whole
engineering team since the beginning. The whole org is on board with this
change. It's definitely not happening in a silo.

> This is definitely the second system effect. It's going to take years to
> make this thing reliable and sunset the old product.

I hope not, but obviously we're not done yet, so I can't say how long it will
end up taking to completely decommission the Python 2 app. What I can say is
this: there are aspects to this project that are _simplifying_ our system and,
for what's left moving from Python to Go, our intention is to port the
business logic as close to a straight up port as we can get.

> \- They're porting over the logic? Even if this is actually the right move,
> wouldn't a clean-room implementation potentially give better outcomes?

_That's_ second system effect, to me. We can't change everything and fix every
problem now, so we're focusing on the changes that will help us move from
Python to Go faster.

> \- Why are they continuing to use App Engine if the writing's on the wall
> for 2024?

I don't think Google Cloud is disappearing in 2024, for one. Beyond that,
again, we're not changing everything about our architecture. The way our data
is stored is staying the same.

~~~
scottrogowski
> The whole org is on board with this change. It's definitely not happening in
> a silo.

Given the seemingly strong chorus of voices responding with cautionary tales
about why you might want to rethink this plan, and the number of engineers in
your organization, it seems more likely that you have some dissenting voices
who have either been too scared to speak up or have already been shot down.

~~~
dangoor
What I meant by "the whole org is on board" isn't that there weren't other
opinions. There have been multiple opinions on almost every decision we make
(and we have an open process and document our decisions in a style very much
like this one[1]). In the end, it's not about "shooting down" alternatives,
since that's loaded language. It's about making what we think is the best
choice we can with the information available to us.

Even in this thread, there's a chorus of voices sounding caution based on
their limited information of what our situation looks like, but there are
others who see why we're doing this, based on the same limited information.

We absolutely do know the risks of this project, which is why we're doing this
as incrementally as possible.

[1]: [http://thinkrelevance.com/blog/2011/11/15/documenting-
archit...](http://thinkrelevance.com/blog/2011/11/15/documenting-architecture-
decisions)

------
kucing
Looking at the case where khanacademy is migrating their server only after
about 10 years. I realize more that I don't have to worry that much about
being locked into certain technologies (unless it's clearly untransferable,
e.g. storing part of customer data in 3rd party server), because after all, we
might keep it only for about 10-20 years, and the thing I'm working at almost
certainly will only last < 2-3 years.

------
todd3834
> We’ll only generate web pages via React server side rendering, eliminating
> the Jinja server-side templating we’ve been using

I’ve been down this road. Deep down this road. Let me just give you a heads up
on something I didn’t consider at the time: Most template languages do not
parse every single node, one by one. In a sense they are just doing string
concatenation. Not so with server side rendering and React. I’m not saying it
can’t be done but just realize it is going to take a lot more compute power.
Caching is great of course but won’t help you if you plan to customize user
content during the server side rendering as well. My recommendation is that
you don’t do any user authenticated stuff during SSR.

Also consider how you are going to handle cookies if you do plan to make
authenticated requests to server side rendering. Also solvable but for some
reason people had the hardest time understanding why we had to forward cookies
to the domains we controlled in an API request and definitely not to any other
servers.

I’m not sure I would pick React for an SEO driven website. It is hard to get a
competitive “time to first byte”. Unless of course you can pre warm a cache of
every one of your pages.

Lastly, you’re going to need Node for the SSR. I’m sure you know this but that
might take you out of app engine and into cloud compute. Not a big deal but
thought I’d mention.

Good luck! It is doable. If you ever want to chat about how we solved some of
these problems I’d love to save you some time if I can. Hit me up in my
profile email.

~~~
dangoor
Thanks for the suggestions and the offer to chat!

We've been doing SSR for quite a while now and are improving our CDN use as we
go along. We already took steps to ensure that there's no user-specific
information showing up in our server-side react rendering which would damage
cacheability.

Our frontend infrastructure team essentially owns the React render server.
I'll let them know you offered to chat.

------
sethammons
I've been in a similar boat. We've been splitting up or converting large
Python 2.6/2.7 applications into Go services (and doing the same to large Perl
applications) for a long time now.

Go has consistently been 10-20x performant (allowing for dramatically reduced
hardware needs), easier to maintain, and more productive to produce code in
than our previous Python (Twisted) and Perl (AnyEvent).

Hopefully KhanAcademy has solid telemetry data in both legacy and new code so
they can quantify benefits. They will also have a learning curve for managing
multiple micro services vs monoliths. Accessing shared data will be a problem
they will likely have to solve. We've opted for each service controlling its
own data - no reaching into another service's data behind its back. Everything
through APIs. This gives the microservice the ability to alter its datastore
as it needs to and not be blocked by other teams' need to update how they
access the data.

Debugging a distributed solution is much harder than a single service.
Distributed tracing, consistent structured logging with log aggregators that
let you do fancy searches (like Splunk), and application telemetry and metrics
will be even more important than before.

~~~
dangoor
Does sound similar!

We have established the rule that each service owns its own data.

We've already got Stackdriver set up to give us distributed tracing and have
set up standards around logging.

------
benatkin
The article says this: "Moving from Python 2 to 3 is not an easy task."

I disagree with this. It's a Python project's dependencies that make it hard
to move from 2 to 3, and most libraries have been updated.

Of course, you could argue that it isn't easy to migrate a codebase from one
major version of a language (or framework, or database) to another, but when
you eliminate _easy_ from your vocabulary it becomes harder to describe
different levels of difficulty.

~~~
foolfoolz
migrating from python 2 to 3 is such a large task that migrating to any other
language is a comparable effort. this is not just a library problem the
language itself changed significantly

source: no python services at my company are going to be migrated to python 3;
it’s all moving to a JVM

~~~
CivBase
> migrating from python 2 to 3 is such a large task that migrating to any
> other language is a comparable effort

I'm going to call BS on that one.

If you're having issues with Python 2, then it might make more sense to switch
to another language instead of upgrade to Python 3. But going from Python 2 to
3 is _much_ easier than switching languages completely.

Python is not a perfect language. _There is no perfect language_. It sounds
like your company just had a reason for switching to a JVM language and the
Python 2 EOL was a justification to start.

~~~
glofish
I agree - the statement that migrating from Python 2 to Python 3 is a
comparable effort to migrating from Python 2 to Go feels grossly exaggerated.

~~~
tapirl
AT least, it is more exciting for a developer to migrate from Python 2 to Go
than to Python 3. :D

------
d_burfoot
I heard about this project from a friend who works at KA. I am concerned about
the strategy, and I think the following approach would yield better results:

1\. Write in Go an exact reimplementation of the current Python codebase. Use
the same database schema, front-end HTML/JS, test suite, and so on. To
whatever extent possible, use the same names for classes and functions. Check
the reimplementation correctness by using a comparison tool that calls both
the Python and Go version of a page/function/search and making sure that they
produce the same results.

2\. Change the production code over to the Go version, perhaps using a ramping
strategy where X% of servers are running the Go code, and you gradually
increase X, while monitoring vital statistics like server load and response
time.

3\. Now that the production site is running Go, incrementally split off
components into their own services.

This approach leads you to the same destination, but with a lot less risk. It
is very unhealthy to have a situation where the production site is running one
codebase but all the developers are working on another codebase. Note that you
will realize the benefits of Go (performance, type safety) after step 2, which
is much sooner than OP's plan.

Joel Spolsky's classic essay about how you should never do full codebase
rewrites is worth reviewing:

[https://www.joelonsoftware.com/2000/04/06/things-you-
should-...](https://www.joelonsoftware.com/2000/04/06/things-you-should-never-
do-part-i/)

~~~
krenel
100% agree. We've just finished to roll out our implementation in Go migrating
a subsystem from PHP and receiving around 150req/second and demultiplexing
those request to 1500-2000req/second to legacy backends.

The key to the success of the project was that the API was an exact match, and
we could compare both implementations for exact requests. The deploy strategy
of the new version:

\- Reply the real traffic to the new Go service comparing the results with the
old one \- Then implement a toggle feature than enabled different traffic
sources to use one backend or the other \- Keep changing backends to the new
system and ensure that metrics were unaffected

Having e2e and integration tests for the Golang project was of a huge help,
since we could fix all differences using TDD.

Although we changed some of the implementations to take advantage of Go
constructs, just a 1-to-1 replacement would have had a huge performance
impact.

------
wonjohnchoi
It is crazy to see they are still using python 2. Seeing how slow the
conversions to python 3 have been, was creating python 3 a good decision for
python community? Can it be argued that developing python 2 further in a
backward compatible way would have been better for the community? I know that
evaluating this kind of thing is hard as metrics are bound to be subjective
and speculative. But I am curious if there was any serious attempt to figure
it out.

~~~
speedplane
> was creating python 3 a good decision for python community? ... I know that
> evaluating this kind of thing is hard

I don't think there is any question here: Python 3 is a complete disaster.
Years and years of engineering effort wasted on changing string libraries.
Sadly, the Python leadership refuses to acknowledge the failing, perhaps
because such an acknowledgement would challenge their omnipotence ... it
would, and it should.

------
aazaa
> Now, in 2019, Python 3 versions are dominant and the Python Software
> Foundation has said that Python 2 reaches its official end-of-life on
> January 1, 2020 , so that they can focus their limited time fully on the
> future. Undoubtedly, there are still millions of lines of Python 2 out
> there, but the truth is undeniable: Python 2 is on its way out.

The Python 2/3 split is by far the most annoying thing about Python. I don't
develop software in Python but about half the time I've had to use a Python
library or program the problem of 2/3 incompatibility has cropped up. Some
projects don't make it clear whether one or the other is required, leading to
further confusion.

If anything the Python 2 EOL could make a bad situation worse. Like Khan
Academy, each Python 2 package maintainer will be forced to make a decision:
move to Python 3 or abandon and maybe move to an entirely new language. It
think many will choose to abandon, leaving these packages to rot.

Second on the list are the multiple package managers (or things looking like
package managers).

Third on the list of annoyances are native extensions, driven by the poor
performance of Python itself. These extensions make it difficult to use
certain libraries across operating systems.

So as a non-Python developer I don't look forward to the occasions when I must
use a Python-based piece of software.

~~~
zo1
If you're installing a package via the package manager, it will very quickly
tell you if you can install it on your specific version of python. Unless
you're downloading some rather obscure and un-loved library where the author
didn't explicitly state which versions of python they support.

Multiple package managers: There has only been 2 big ones from my long-term
general usage of python. Easy-install and pip, the former of which is falling
in favor but still semi-supported. Pretty much everything runs off of pip.
What may be confusing you, and does confuse me at times as well, is their
naming and the installation instructions as provided by library authors. E.g.
some say python setup.py -install others just tell you to "pip install" it.
Some would say use "setuptools", etc. Other would tell you to use things such
as "conda" or "anaconda", pipx, and to create virtualenvs. All secondary, but
things that should not ideally distract you from just plain using pip.

3\. This has also been getting a whole lot better in the last 5 or so years.
Microsoft has been funding dev-time to make the ecosystem for python
(including extension compilation) much more pleasant in the Windows space.
Also, the package managers and library authors are doing a whole lot better in
that binary distributions are much more prominent so the compilation of the
extensions never has to happen on your machine.

~~~
saagarjha
> Unless you're downloading some rather obscure and un-loved library where the
> author didn't explicitly state which versions of python they support.

I take it that you've never tried the obscure and un-loved pwntools package :)

------
atmosx
How does Khan academy make money to sustain the website? Is it all donations?

~~~
bhaumik
Non-profit, with several $1m+ donations:
[https://www.khanacademy.org/about/our-
supporters](https://www.khanacademy.org/about/our-supporters)

~~~
patneedham
To be more specific (for anyone else who hasn't checked the link yet), the
"Lifetime giving" section has: 9 donations >10m, 4 donations between 5-10m, 20
donations between 1-5m

So looks like there has been at least 120m in donations!

~~~
atmosx
Thanks

------
tybit
As much as I personally don’t enjoy writing Go I really can’t fault them.

I still find it interesting that for a relatively obvious feature set of fast
compiles, fast startup and fast runtime there really isn’t anything mainstream
out there to compete with Go.

I really hope something like Kotlin, Swift, ReasonML or even AOT JVM/.NET
brings something to the table soon. Or perhaps I’ll just have to wait for WASM
to really take off server side.

~~~
pushrax
How does WASM fill this gap?

> fast compiles, fast startup and fast runtime ... mainstream

C is this but it's harder to write secure/correct C, the standard library is
smaller, and there's no canonical toolchain in the same way as Go.

~~~
PudgePacket
One doesn't just write WASM though.. WASM is like the JVM. You still need to
write code in some other language.

~~~
pushrax
Exactly. Hence posing the question (rhetorically, I suppose).

------
timwaagh
Seems like such a waste. Is switching to python 3 really that hard? Is
hardware that expensive? If this is indeed the right call it doesn't bode well
for traditional scripting languages as the web scales to fewer high traffic
apps. We might start to see more jvm, go (apparently) or even rust and c(++),
rather than speed of development languages like Python or Ruby. Trend seems to
be the reverse though, with python the second and most rapidly growing
language.

~~~
neurobashing
Everyone’s project/code base is different but in my experience there’s been a
critical mass of libraries for a few years. I presume the “it’s hard to move
to 3” is dev teams wanting a new toy as much as “the rewrite is too complex”.
Library use, size of code base etc are all big factors but at the end of the
day, I think team motivation is really the deciding factor.

~~~
freyr
That mirrors my experience as well. Someone with influence is bored or wants
to level up, so they'll drag the entire company into a long, expensive
quagmire.

Unless the existing codebase is mired in technical debt and completely
unsalvageable or cannot scale further, this seems like a very radical move.

~~~
lonelappde
Replacing an old unholy mess with a new unholy mess is usually a bad plan.
It's called the Second System Effect.

------
tmpfile
Yikes! It's a lot of effort to reduce memory use. They might be better off
creating a new Go entrypoint/server that can call into CPython to reuse all
their existing/tested modules (treat their Python as a microservice called by
Go). They could then use Go to create/call new microservices or replace
various routes on a selective basis.

------
mjpuser
I think the real problem is that they didn't properly maintain their code.
Rewriting it in Go won't prevent them from dealing with this in a few years
for when this Go version reaches end of life. I would have liked to see an
article on "introducing process" side of programming.

~~~
dangoor
Thanks for the comment! A couple of things about this...

1\. the Go team is working very hard to ensure that there are no such
compatibility issues. Code written for Go 1.0 should still compile with Go
1.14 beta today.

2\. It's possible there's more we could have done along the way, and I tend to
think that statically typed languages make it easier to safely refactor more
ruthlessly. But I do think we've actually done quite a bit of change
incrementally along the way. Our move to React on the frontend and GraphQL on
the backend have been good examples of that. Plus, we did a huge refactoring a
couple of years ago to draw better boundaries in our monolith, and that has
made a move to services possible.

------
ngngngng
Unpopular opinion, writing Go is faster than Python. With the compiler, strong
typing, and no versioning hell, I'm much more productive in Go.

Whenever I use python I run into problems with versions and dependencies. And
the whole community just tells me to use pyenv or virtualenv and it will "fix
all my issues". Only it doesn't.

~~~
kart23
For larger stuff, yes. For small stuff, no. If you just need to get something
small done quick and dirty, python will be easier and faster.

~~~
jerf
Yes. My current estimate of the cutover, for myself, is about three weeks of
solid, 40hour/week development. After that, my Python (or other dynamic
language) starts the process of seizing up, where instead of rewriting some
module I just put a little hack in there to make it backwards compatible with
other code, since I haven't got a great way of being quite sure what's calling
this code, so I use a __setitem__ or have a function that takes "a thing or an
array of that thing", and I find myself increasingly reluctant to refactor the
Python.

YMMV on the exact number, but that's been my experience several times now.

I know it can be done; I've seen it done, I've done it myself. But refactoring
without even the rudimentary static type system Go has just becomes an
increasing nightmare at scale.

And I use unit testing in Python, etc.

But, flipside, yes, Go isn't a great language for just bashing a script
together in. Maybe not the worst, with a bit of library work, but not a great
language.

------
bootlooped
The article mentions Go's superior compile time, when compared to Kotlin. I
have done a lot more Java development than Kotlin, but my recollection is that
both of them compiled fairly fast.

Is Go really significantly faster to compile for similarly sized projects?

~~~
tbrock
I’ve written some things in both languages and compiling go feels an order of
magnitude faster.

~~~
abraxas
It's mostly because most setups will use maven or gradle which do much more
than just compile code. They will often pull down dependencies, check for bugs
and style inconsistencies, run unit tests etc. If you run plain javac against
a bunch of files it will compile just as fast as Go. In any case it will be in
a blink of an eye.

------
remote_phone
What a mistake.

Not only going from a monolith to microservices but also changing the
language? This is a mistake that rookies make. This will be one of those post-
Mortems where they will sheepishly admit they bit more than they could chew,
and it wasted years of productivity.

There’s no reason to move to Go. Stick with Python for now. Migrate safely to
python 3. Once everything is stable, start breaking things up into thrift or
protobuf services. They don’t even need to be microservices but you need the
contract. Once that is stable migrate to whatever language you want. But at
this point you will have the well-defined api and test cases. Trying to do too
much all at once is a no-brained disaster.

~~~
pjmlp
This kind of decisions is usually always taken when the monetary cost is not
measured.

Apparently not mapping developer hours to real money keeps not being a thing
outside consulting shops.

------
sibeliuss
Isn't there a quote somewhere along the lines of "full rewrites are suicidal?"

Seems pretty risky to me.

~~~
dangoor
You might be thinking of this excellent article by Joel Spolsky:
[https://www.joelonsoftware.com/2000/04/06/things-you-
should-...](https://www.joelonsoftware.com/2000/04/06/things-you-should-never-
do-part-i/)

(Ironically, that post is about Netscape and the rewrite, Mozilla, ended up
growing as a nonprofit far beyond Netscape's original scale.)

The so-risky-it's-almost-certain-to-fail approach is a big bang rewrite. Stop
the world until the rewrite is done.

That's not what we're doing. A tiny bit of our GraphQL schema is _already_
running in production atop new services written in Go. We're going to do this
step-by-step, while keeping the site running, _and_ adding committed features
next year.

It's still a huge investment and comes with risks, but it's an incremental
process and we'll be able to track the progress at every step.

~~~
mntmoss
So long as your data has well-defined interface boundaries on it, this is a
good way to de-risk rewrites. It really depends on the current state of the
codebase. If there's a tangle of dependencies, then they have to be cleared
out or facaded away to a testable interface, or else that part of the port
will be a shot in the dark.

And if you have the interfaces in hand, the language itself becomes
considerably less important, since more of the code is subsequently dependent
on your own system and not really the outer ecosystem. It's the projects that
have coded "to the metal" on their existing platform that have the biggest
issues with keeping up their flexibility.

~~~
dangoor
Given that this is mostly about our backend systems, we've got one big
interface in front: our GraphQL schema.

Our system isn't perfect, but a couple of years ago we spent a good deal of
effort detangling our monolith. This plan wouldn't have been an option had we
not done that.

------
tony
You can start porting your live codebase to python 3 right now - no downtime
at all. You can have a 2+3 compatible codebase live in a week or two, it'll
get you about 98% the way there [1]

My projects (e.g. [https://tmuxp.git-pull.com](https://tmuxp.git-pull.com),
[https://unihan-etl.git-pull.com](https://unihan-etl.git-pull.com)) are both
python 2/3 compatible.

I learned by reading through other projects like Sphinx, Flask, Werkzeug, and
SQLAlchemy:

\- Flask: [https://github.com/pallets/flask](https://github.com/pallets/flask)

\- Werkzeug:
[https://github.com/pallets/werkzeug](https://github.com/pallets/werkzeug)

\- Sphinx [https://github.com/sphinx-
doc/sphinx/tree/v1.8.5](https://github.com/sphinx-doc/sphinx/tree/v1.8.5) (see
compat: [https://github.com/sphinx-
doc/sphinx/blob/v1.8.5/sphinx/util...](https://github.com/sphinx-
doc/sphinx/blob/v1.8.5/sphinx/util/compat.py))

\-
[https://github.com/sqlalchemy/sqlalchemy](https://github.com/sqlalchemy/sqlalchemy)

Helpful blog posts: [http://lucumr.pocoo.org/2011/1/22/forwards-compatible-
python...](http://lucumr.pocoo.org/2011/1/22/forwards-compatible-python/),
[http://lucumr.pocoo.org/2013/5/21/porting-to-
python-3-redux/](http://lucumr.pocoo.org/2013/5/21/porting-to-python-3-redux/)

As for automation tools, for 2/3 compatibility, I used futurize and huge
codebase to great success: [https://python-
future.org/futurize.html](https://python-future.org/futurize.html). I started
with this:

    
    
        futurize --write --stage1 --unicode-literals --nobackups <files>
    

and
[https://docs.python.org/2/library/2to3.html](https://docs.python.org/2/library/2to3.html)

    
    
        2to3-2.7 -w -n <files>
    

Really helped.

Also, wire Travis / your CI system to have your tests run on python 2 and 3.

If you want to test your python 3 codebase live on a subdomain / same
SQL/NoSQL DB, be careful about jobs/tasks! Pickle version mismatches and
stuff. Use a separate redis/whatever DB for the deployments.

After you're finally on python 3, you can use
[https://github.com/asottile/pyupgrade](https://github.com/asottile/pyupgrade)
to modern your code.

[1] For the other 2%, use conditionals in your requirements file:

    
    
        enum34==1.0.4;python_version<"3"
    
        ushlex==0.99.1;python_version<"3"

~~~
dangoor
I appreciate all of the links and the fact that, for a lot of folks, this is
the state of Python 2 to 3 migration today. What you're suggesting is, imho,
the best path for most (and I appreciate you mentioning the pickle
incompatibility because that is a thing which would trip people up when trying
to make the move).

If we would have been able to have a 2&3 compatible codebase live in a week or
two, we absolutely would have done that. Incompatible changes in some
libraries we use, App Engine first gen to second gen changes (which are for
the better, but still a big deal), the choice of storing some pickles
permanently, plus a need to really verify unicode handling all over the place
(especially in a 10 year old codebase), and other factors that aren't coming
to mind at the moment mean that this is not a couple week thing for us.

Moving to Go is more work than moving to Python 3, but in our particular case
it's not as much more work as people might expect.

~~~
tony
Thanks for your reply to my reply! Sure, it may take more than two weeks, and
maybe golang is a good choice for your stack. Maybe a better way of saying it
is python 2/3 compatibility can be gradually adopted while maintaining.

Also, I can't speak for the technical specifics of your project, but I would
like to speak a bit from my prior experiences:

> Incompatible changes in some libraries we use

What are those libraries? There may be python2/3 forks available on pypi. For
instance, on Peergrade we went from pyPdf -> pyPdf2, boto -> boto3 (that one
is a _lot_ of work). We were still able to stick on the python 2 codebase, but
gain python 3 forward compatibility.

In the case of very specific packages, we had to do forks. We had to move some
patches from a python 2 only library and port them into a python 3 only
library, then do one of those version constraint things.

> App Engine first gen to second gen changes (which are for the better, but
> still a big deal)

I'm not familiar with app engine so can't speak to it.

> the choice of storing some pickles permanently

Can you clarify this?

I can't speak for it without seeing it, but would it be possible to serialize
the data to json, use python import strings
([https://devel.tech/tips/n/djms3tTe/how-django-uses-
deferred-...](https://devel.tech/tips/n/djms3tTe/how-django-uses-deferred-
imports-to-scale/)), then write a migration to port the old pickles over, and
have something more portable?

It's possible, depending on what's being stored, it'd also help you migrate to
golang if the data stored in it could be consumed by a go service.

> plus a need to really verify unicode handling all over the place (especially
> in a 10 year old codebase)

Yes I had a lot of pitfalls with this one. Areas to look out for are hashing
functions. They are very strict in whether they're dealing with bytes or
strings.

Are you already using unicode_literals? Those can be implemented gradually in
a current codebase.

How is your test suite? Sometimes having test coverage, even naively, can also
act as a smoke test to catch unicode issues.

Selenium tests can get a lot of coverage for very cheap to verify behavior at
a high level.

> Moving to Go is more work than moving to Python 3, but in our particular
> case it's not as much more work as people might expect.

I think the services + golang part is smart, and can't speak for the specifics
of your codebase.

I will say in hindsight, I feel the effort / hours I put into upgrading a
Python 2 codebase -> 2/3 and eventually 3-only made it _much_ easier to
breath. Cleaner syntax, no unicode headaches, and no need to have the
lingering prospect of a language exodus looming over the head.

One negative aspect we haven't talked about those with "gradual" python 2/3
migrating is breakages. There are refactors that end up being done that when
pushed out risk breaking - and when they're purely internal code changes.
There is a business case to eliminate the tech debt because the value
equation: A python 2 codebase, even with warts to it, is meeting an EOL in the
next few days ([https://pythonclock.org/](https://pythonclock.org/)).

Even assuming golang + microservices is the final destination. Amortized, on
the projects I've been on, moving to python 2/3 (or 3-only) paid back. If
there's opportunities where the python codebase could be split into apps /
separate wsgi entry points, then have golang services replace them later -
_that_ could be an option. Even without golang, the (probably huge?) refactors
involved in moving to python 3 and being "(micro?)service ready" has benefits.

~~~
dangoor
We did do a bunch of refactoring a couple of years ago to draw boundaries
within our monolith. We almost certainly wouldn't have made this choice had we
not done that work. We're also doing work in Python to smooth the way for this
change. For example, we're migrating our remaining REST endpoints to GraphQL
in Python before making the move to Go.

Regarding the pickles: we are essentially doing what you're suggesting. We're
going to migrate them to JSON. This is one of those things that is going to be
a pain to do and we'd have to do it whether we're switching to Python 3 or Go.

Also, I'll note that the Python 2 EOL is only half real. There's so much
Python 2 out there that, while the PSF is no longer supporting it, there will
be people supporting it as needed.

I do agree with you about Python 3 being much cleaner. But we have to do
significant rewriting of our devserver, a lot of our data access code, and
some other libraries that I don't remember offhand. Again, it's quite specific
to our codebase. People with a "normal" Django app are unlikely to have such
issues.

------
thatswrong0
Ok this is going to sound ignorant, as my only experiences in backend services
have been Go and Python. I don't like either. Is there something I'm missing?
For simple CRUD apps, both are sufficient. But (in my limited experience), the
moment I've wanted to create more complex business logic with stricter
constraints, neither has been quite up to the task.

Go doesn't make things easy. It asks you to repeat yourself. I don't like the
lack of basic functions like a generic map / filter function.. I know Rob Pike
just says "use for loops", but it feels so unnecessarily unexpressive. When I
see map, I know what's going on almost immediately. For loops take more
reading to understand. Nil pointers shouldn't be, yet still are, a thing
(developers aren't perfect - why can't the type system help?). It feels like a
straight downgrade from Python from a code clarity perspective. And it's
typed, sure, but the type system doesn't let me express constraints that other
languages allow me to do that would prevent entire classes of bugs. It doesn't
feel worth it compared to Python. Yes, the core language ends up being
comparatively "simple", but simple building blocks doesn't guarantee a simple
overall system. And my company is very diligent from an architectural
perspective.

But then when I look at Python, I'd rather just use Javascript with Lodash,
esp. when it comes to the treatment of functions as first class objects[0].
Throw Typescript in there, and you get, in my opinion, a better type system
than Go, so unless language performance is a major constraint (which it hasn't
been for my company, our DB usage patterns it the biggest thing instead), why
would I want to use either of these rather than Typescript?

[0] Edit: Dumb and wrong, I meant its treatment of anonymous functions. I
don’t like lambdas.

~~~
catalogia
Go seems to be optimized for onboarding new developers (particularly straight
out of school) quickly, rather than for the long term comfort of developers
using it. There are Rob Pike quotes that speak to the first part of that at
least.

If I was a business owner, I'd love Go. But what I can't really figure out is
why so many devs love it. It's far from the worst thing in the world, I don't
hate it, but I just can't get excited over it either.

~~~
dx87
From what I can tell by reading what Go developers write, they like it
precisely because it's not something to get excited over. It seems like the
kind of language where once you learn it, you don't have to keep up with a
bunch of blog posts detailing all the cool new things being added to it, and
decisions over stuff like formatting are made for you. I primarily use Rust
for hobby projects, and the steady stream of new features/libraries can get
tiring. You don't have to use the new stuff in your own code, but if you want
to use popular libraries, you'll probably be stuck using the new features.
Making everything async seems to be the new hotness, and sometimes I wish the
language would be nothing to get excited over.

~~~
SomaticPirate
It also lends itself to patterns in code. One thing that learned about Python
coming from Perl was there was a “Pythonic” solution. Go seems to take that
idea further with gofmt and a small but powerful std lib. I can typically
glance at how someone is configuring their http.Server and understand the
intention of the code.

~~~
idoubtit
The "pythonic" way was a mantra that has been obsolete for a long time. With
Python, there are many (unsatisfying) ways to declare dependencies, many ways
to format code (I hope "black" will prevail), many ways to format a string,
many ways to create a struct/record...

The lack of quality batteries included also leads to "there is more than one
way to do it" through the choice of unofficial libraries. My experience is
that the extensive standard library of Go brings more normalisation.

------
buboard
If they were going for performance, and web-related, how come they didn't
consider PHP?

------
Possiblyheroin
I was under the impression that Go was more performant than Kotlin. TMYK :)

~~~
tus88
Go was written for servers, Kotlin for mobile apps. Seems an obvious choice
really.

~~~
hn_throwaway_99
> Go was written for servers, Kotlin for mobile apps.

Not really. Kotlin was just a better JVM language developed by the JetBrains
folks before Google adopted it as a first class Android language, but I don't
believe it was specifically developed for mobile initially.

~~~
dehrmann
The open question is if Java will start borrowing the best features from
Kotlin and it will become less relevant (like Scala). To Kotlin's credit, it
has better IDE support, it feels simpler than Java (whereas Scala feels more
complex), and it cleans up a lot of fundamental language issues that Java
can't get away from.

------
Thaxll
Kotlin looks nice now but on the long term it's a bad choice since they're
stuck on Java 8 they will always lag behind the real JVM and won't be able to
catch up since they need heavy modifications.

~~~
m0shen
If you mean the java 8 bytecode version, that hasn't been true since April:

[https://blog.jetbrains.com/kotlin/2019/04/kotlin-1-3-30-rele...](https://blog.jetbrains.com/kotlin/2019/04/kotlin-1-3-30-released/#more-6991)

~~~
Thaxll
But they don't benefit from new runtime features right ? Loom, GC ect ... are
no go for Kotlin.

~~~
m0shen
I don't see why it wouldn't benefit from the GC changes. AFAIK Loom has no
scheduled release date.

They seem like they're committed to supporting new features, but you're
probably right they will lag on some things.

------
Solar19
Interesting read. It always surprises me that companies go many years running
major apps and infrastructure with interpreter-based languages like Python and
Ruby to begin with. It's an incredible waste of energy, compute, and for web
apps sometimes, users' time.

Developers need to be a lot more disciplined about performance and efficiency.
I'm glad Khan went to Go, but man all those years wasted.

