
Disqus: It's Still About Realtime, But Go Demolishes Python - geerlingguy
http://highscalability.com/blog/2014/5/7/update-on-disqus-its-still-about-realtime-but-go-demolishes.html
======
jaegerpicker
While I'm pretty sure that Go is much faster than python in nearly all tasks,
I'm still skeptical that a) they built a full replacement of the system in
only a week and b) that they didn't improve the algorithm in the go code. I
work at a python/django shop and I recently updated a process that could only
handle 5000 transactions in 5 minutes to be able to handle over 100,000 in
just under 3 minutes. This was all in python would go been a lot faster? Sure
but how much is the new code path and how much is the advantage of golang?

~~~
jaegerpicker
Also what to point out that I fully realize that Go is a lot faster than
python at preforming the same task but this kind of analysis is dangerous
because it leads inexperienced engineers to believe that changing languages is
a good performance optimization which should usually be the last step/idea
considered. It's expensive and dangerous to switch languages mid stream and
while it's sometimes the exact cure it needs to be weighed very heavily to
make sure it's the right situation. This article, at least the headline misses
that piece of the puzzle pretty fully IMO.

~~~
blt
Not sure I agree with you there. IMO, porting between languages with similar
abstraction levels is not that hard. I ported some Python code, whose logic
took weeks of fine-tuning, to C# in less than a day. Go still has a garbage
collector and its type system is similar in philosophy to Python's duck
typing. Unless you are porting extra dynamic code (eval(), monkey-patching) or
deeply intertwined with an unusual library, it should be a fast process. On
the other hand, I was leaving Python get real OS threads, not to increase
performance.

~~~
wting
This is nitpicky, but Python threads are kernel threads. It's the GIL that
restricts CPython to a single core.

OTOH goroutines are green threads multiplexed onto kernel threads (aka N:M
thread model).

------
a3n
Go is looking more attractive, but the language is not just the language.

When Go has available a significant fraction (subjectively) of the "batteries
included" that Python has, then I'll start investing time in Go. Is it there
yet (subjectively)?

~~~
wtbob
I think Go is pretty competitive with Python's standard library. It doesn't
have an FTP client (we use and are happy with github.com/jlaffaye/ftp) or an
equivalent to imaplib (although github.com/mxk/go-imap/imap looks
interesting), but for the rest of our workload Go has everything we need.

~~~
daave
Aside: for SFTP we use (and like)
[https://github.com/minusnine/gosftp/](https://github.com/minusnine/gosftp/)

------
peterwwillis
> In only a week a replacement system was built

This part is telling to me. It means their codebase isn't so complex, and
optimizing their application for their hardware would take as much time (or
less) than adding hardware. For people with large, complex codebases, scaling
horizontally increases capacity much faster.

------
wting
In the past Go has had trouble with garbage collection lag due to a global
mark and sweep implementation[0]. Was this not an issue with Disqus's new
implementation?

[0] [https://groups.google.com/forum/m/#!msg/golang-
nuts/S9goEGuo...](https://groups.google.com/forum/m/#!msg/golang-
nuts/S9goEGuoMRM/FZyd2M6uiVMJ)

For the record Python uses a generational garbage collector.

~~~
devanti
You can run multiple instances of the same Go program to reduce the effect of
GC lag

------
awda
> yields it's own benefits as well.

yields _its_ own benefits as well.

------
infogulch
I was going to ask how pypy performed on their original python code but it
looks like gevent doesn't support it yet (or didn't at the time).

------
Oculus
_Node was not selected because it does not handle CPU intensive tasks well_

I'm curious to hear what sort of CPU intensive tasks Disqus does.

------
axelf
why were they using python for performance critical code in the first place?
Go seems closer to java's niche to me.

~~~
sologoub
Disqus was originally built on Django:
[http://blog.disqus.com/post/62187806135/scaling-django-
to-8-...](http://blog.disqus.com/post/62187806135/scaling-django-to-8-billion-
page-views)

It seems that this is what the found team was most comfortable with, so it
makes sense that they proceeded to solve problems using tools they already
knew well. At some point, they exhausted how far they could take their
existing tools and started investing into new tools.

~~~
melling
What Go web framework did they move to?

~~~
voidlogic
It doesn't say, but probably none. My guess would be they just used "net/http"
\+ Gorilla (maybe). Gorilla:
[http://www.gorillatoolkit.org/](http://www.gorillatoolkit.org/)

~~~
elithrar
Given it sounds like this application is not "web facing" (i.e. not an API nor
rendering HTML), the use of any "web framework" or Gorilla doesn't make much
sense.

~~~
voidlogic
I write back-end stuff like this all the time and it often has admin
interfaces.

