
How We Went from 30 Servers to 2: Go - treeder
http://blog.iron.io/2013/03/how-we-went-from-30-servers-to-2-go.html
======
zobzu
It's not really "go" that makes the difference. it's how the runtimes and
frameworks are used and/or made.

frameworks on top of frameworks, all being over engineered, with poor
understanding of what the system actually does, result in super slow apps on
top, that you generally go to aws to scale.

It's not the first time that I see people reducing a dozen servers that were
"always maxed out" by a couple of servers "that can barely feel the load".

Yeah throwing hardware at the issues is fine'n all but we're been way over the
limit too many times in too many directions.

Go provides a clear start/APIs/framework, and the language enforces good
habits. Also, it doesnt have things like global interpreter locks.

The lesson? Stop using cool techs because there's blogs about them. UNDERSTAND
the tech before using it.

~~~
rektide
I'm very sympathetic to your tech-on-tech-on-tech-on-tech hogwashery
promulgated by open sourcers & vendors prosteltyzing their wonderful solutions
and the other acculturation factors that permits developers approaches other
than trying & seeing & exploring-

That said, you are wrong.

I'm not a Go user and I don't care for it, but Go is fundamentally different
and better than most runtimes. Go has it's own green threading, which allows
it to start new tasks using extremely small amounts of memory, and it can
switch between tasks without the normal cost of context switching full on
threads.

That is a huge difference. Node has some elements in play here, with it's
event loop and callbacks, but really the only thing in the field that compares
to Go that's in use is Erlang, and there's no frameworking on top of Erlang
that wouldn't cause most PHP or nine tenths the Python developers to curl into
a weeping ball at the Eldritch horrors they'd been exposed to: even ASM code
is not nearly so effective, being clearly a low level somewhat harmless thing.

Being able to kick off small processes, that can sit around for a really long
time, awaiting data- that's a model of computation we haven't done in decades,
never with the mass popularity- there's a TLA for this kind of processing-
_Communication Sequential Processes,_ and Go is it. It's entirely different
from the procedural model, from the Python and Perl and PHP scripting, where
concurrency is a carefully waded into thing- Go is a concurrent first runtime,
about ongoing, enduring, concurrent processes, and that's huge, _HUGE_ I say,
huge.

~~~
nkohari
_> I'm very sympathetic to your tech-on-tech-on-tech-on-tech hogwashery
promulgated by open sourcers & vendors prosteltyzing their wonderful solutions
and the other acculturation factors that permits developers approaches other
than trying & seeing & exploring-_

Are you studying for the SATs, or just trying to make your argument sound more
compelling through the use of unnecessarily complex words?

~~~
rektide
It's how I feel, I'm so very very sorry it's tripped your acceptable tolerance
levels. I'll bite my lip and not add "you prat" to the end. Thanks for your
considered moderation, and I see your point.

~~~
nkohari
My only point is that being obtuse is not the best way to get your point
across. I assume your goal in expressing your opinion is to convince others,
and to do so it helps to be clear, concise, and direct.

~~~
porker
That's a very narrow assumption as to why someone would express their opinion.
The OP might be sharing their experience (in what was a humorous way) purely
to be sociable in this discussion...

Not everyone is out to convince others.

~~~
mrgreenfur
Righto, he could be here to make himself feel good by using words he doesn't
usually get to.

~~~
rektide
Dare I call into question what you are here for? How do you feel about your
contribution- attempting to heap more shame- plays for the whole audience? Is
what you are doing of value?

------
waterside81
Go's definitely picking up momentum. I know that Mozilla is shifting much of
its services infrastructure to Go.

We're porting over our Arabic sentiment engine, currently in Python/Cython to
Go. If you're dealing with simple data structures, going from Python to Go is
almost a line-for-line port, but the performance benefits are, of course,
massive. Our benchmarks show a 40x speed improvement so far.

Lastly, for anyone thinking about taking the plunge, use Go tip (from their
source code repo) - don't use their "stable" releases. They fix bugs so fast
you'll always want to be current.

~~~
DASD
Do you have a citation for Mozilla using Go? I would be a bit surprised given
their(Mozilla) development of Rust.

~~~
azakai
Rust and Go are not similar or competing. They are both new, but aside from
that the differences are huge. So if a project uses one, that doesn't mean the
other was a possibility for that project too.

~~~
jbooth
They're not similar at all, but both were designed to replace C++. That sort
of means that they're competing for the use-case of "things I would write in
C++ if it wasn't a god-awful nightmare to do so".

From there, though, you're right that they take very, very different
approaches.

~~~
azakai
> They're not similar at all, but both were designed to replace C++.

In practice, the main use for Go appears to be like this story - an
alternative to RoR, Node, Python/Django, etc. - and not a C++ alternative. Go
and Rust may both start from the idea of doing C++ or something similar
"right", but end up in very different places.

------
desireco42
I went briefly to through the comments, it seems that no-one is bothered with
lack of details in this post, things are not logical at all, almost like it
was written by PR agency in charge of promoting Go.

Except it is established business so I would assume things are true, but real
motives are hidden. Go is obviously good language for some specific things,
but it's not like ruby is pure trash, how come you did 'everything' in ruby,
and then didn't know about eventmachine, but just had to rewrite things in Go.

And rewrite happened overnight, right?

A lot is missing from this story. I will definitely look more into Go, mostly
because someone compared it in comments with TurboPascal, and I have fond
memories of Borland tools, Pascal especially.

Go and Rails are so different that there is almost no point in comparing them.

~~~
burntsushi
> but real motives are hidden

Could you elaborate on this please? Are you referring to the OP's motives of
publishing the article? Or of switching to Go?

> Go and Rails are so different that there is almost no point in comparing
> them.

Except when one solves the same problem better than the other.

~~~
lysium
\- They did not say why they did not choose one of the other alternatives.

\- They did not say how they used the concurrent features of Go, only
mentioned that they were there.

\- They did not say how long it took to rewrite.

\- They did not say what they changed in the API.

I'm not saying the story isn't true, but for a true story it lacks a lot. You
could summarize the article with the title and you won't be missing much.

~~~
bascule
They didn't say _why_ Ruby was using a lot of CPU either. Some basic
investigation into the root cause might've revealed some intractable problems
that are tied to Ruby (or at least the MRI interpreter) like long GC pauses
(something which can be mitigated by going to a different interpreter like
JRuby versus a complete rewrite in a different language)

Instead, it's "OMG we're maxing out CPU, time to completely rewrite our app in
a different language"

And how big is this app that it needs 30 servers? We probably serve an order
of magnitude of traffic where I work than these guys do (just guessing) and
don't need that many.

------
trungonnews
I was originally very excited about Go when I first learned about it. But then
I got tired and frustrated quickly after having to listen to the other Gophers
telling me that I don't need this or that feature because there is a better
way to do it in Go.

Like.

I don't need exceptions because Go function can return multiple values.

I don't need a mocking framework like Mockito because Go has interfaces.

I don't need an interactive debugger because I can debug with command line
using gdb.

I don't need named arguments because I can instantiate a struct and call my
function with it. (Have you seen Ruby or Javascript code? Almost every
function takes 'opts' as a single argument. Go is probably going down this
path too.)

Then I learned about Scala. I'm not saying that Scala is better than Go.
However, it has everything that I need. :)

~~~
tptacek
Comments like this are weird. Go does not like exceptions. They've built a
whole idiom around error returns. When you said, "I need exceptions", what did
you expect them to say? "Oh, sorry, we forgot that, we'll get right on it"?

I also don't understand what you mean by "interactive debugger". What is gdb
if not an interactive debugger? Gdb is my go-to debugger for Ruby as well.

~~~
trungonnews
Something like RubyMine IDE debugger or Firebug javascript debugger.

If you are a Ruby developer then I highly recommend trying out RubyMine. I
promise that you will never debug Ruby code using gdb again!

~~~
kragen
I got all excited when I saw your comment and went and checked out the latest
RubyMine video on YouTube, because having something like Firebug for Ruby
would be awesome. But it looks just like what gdb gives you when being driven
by a graphical frontend like DDD or Emacs. Actually it looks a lot less
powerful than DDD. What am I missing?

------
jewel
For large projects, it'd probably make more sense to rewrite the critical code
sections in C. rubyinline makes this really easy, and ruby C extensions aren't
that bad either.

One time on a contracting gig I had some code that was interacting with cairo
drawing and imagemagick. They didn't have a compatible raw image format at the
time (one was RGBA and the other was ABGR, IIRC.) It's trivial to convert
between the two in ruby, but it was taking 60 seconds per image. Once I
switched that one loop to C, it took 0.02 seconds.

I've got nothing against Go, just suggesting an alternative that might make
more sense in some situations.

~~~
dyselon
This definitely does make a lot more sense than throwing out your whole app
and rewriting it in many situations, but I'm always a little hinky about
moving stuff into C just because it has so many ways to shoot yourself in the
foot. You can obviously create bugs in any language (I know I certainly
have!), but Go really does hit a nice spot between safety, productivity, and
speed, so I can definitely see the attraction.

------
stalcottsmith
I've seen a lot of dumb Rails server configs and even dumber usages of Rails
without any tuning at all.

With a couple weeks or a month of work I could shrink the hosting fees or
resources consumed by a factor of 5 out of most Rails apps that are sitting up
around 30 servers... and probably a factor of 10. Leaving aside whatever
rookie or even intermediate mistakes were made in their Ruby code or their
database, this post indicates a lack of understanding of what happened when
their server fell over. Proper tuning of a deployment should not trigger a
100% failure mode like this.

These folks were itching to get off of Ruby for whatever reason... after all
their roots were in Java. If your goal is to do a rewrite and learn a new
language and gain some notoriety why waste time learning what you did wrong
with Ruby or your server config?

~~~
btilly
Your inability to understand their clear description of a basic cascading
failure mode under load speaks poorly of your actual knowledge and experience.
Given that, I have to take everything else that you say with a large helping
of salt.

~~~
ef4
Their description of the root problem was very superficial:

> At some threshold above 50%, our Rails servers would spike up to 100% CPU
> usage and become unresponsive.

Yes, but why? What exactly were those processes doing? Why the sudden change
at a particular threshold?

Their lack of detailed investigation into this makes their post useless to me
-- I have no way of knowing (1) what specific aspect of Ruby's architecture
makes it unfit for their problem?, or (2) is their application doing something
stupid that causes the problem in first place?

~~~
btilly
If you read that sentence in context, your questions are answered. Here are
the previous two sentences for context.

 _The bigger problem was dealing with big traffic spikes. When a big spike in
traffic came in, it created a domino effect that would take take down our
entire cluster_

Given the article to that point, it is clear what is happening. They are
maintaining a steady state of X% (where X is above 50), then a big traffic
spike comes along. That traffic load is not distributed equally (my guess is
because requests are not created equal) and there is a hot spot. The hot spot
fails, then increases the load on everything else. After this repeats a few
times, there is insufficient capacity.

In other words at some steady state threshold above 50%, you wind up without
sufficient capacity to handle traffic spikes. Nothing in this failure scenario
is specific to Rails. It is a well-known failure mode for a cluster under load
with a push based architecture (which http requests are).

The fact that you did not understand that description speaks poorly of your
problem solving skills. Now they may well have been doing something trivial
that is fixable to cause load to be less than it was. But you haven't
convinced me that you're the person to be trusted to figure that out.

------
raphaelj
I'm getting the same experience here by entirely rewriting an old PHP web
application [1] in Haskell using the Yesod web framework.

I've been using JMeter to benchmark both versions of the application. On a
10€/month dedicated server [2], the Haskell one was able to generate 220
dynamic pages per second [3] whereas the PHP one tops at 35 pages per second
on a equivalent page.

Moreover, concurrence capabilities of Haskell are also pretty sweet : while I
was benchmarking the web app using 2,000 concurrent connections, the
application server was only using around 90MiB of RAM. I was not able to
increase the number of concurrent connections as the client application I was
using started to kill my quad core desktop, I suspect Haskell to be able to
manage A LOT MORE concurrent connections as I didn't see any decrease in the
throughput of the application as I was increasing the number of concurrent
connections.

[1] <http://files.getwebb.org>

[2] <http://www.kimsufi.com/fr/>

[3] Static content has been ignored as modern servers like NGinx seem to be
able to carry the static content (CSS, images, ...) at more than 5,000 req/sec
on the same machine.

------
grey-area
Interesting to hear of Go being used in production. It'd be great to hear some
more details on your setup when deploying the go processes - how are you
managing failover, what's your load balancer, and how are you handling
swapping out processes etc? Are you compiling on the server or local machines
before deployment? Most other languages have lots of solutions on the
deployment side now but Go is so new there isn't much info out there.

Also, as you are running an API presumably your app in Rails was pretty
simple, did you have the impression things would be more complex in Go if you
were writing an app with an extensive front end and UI and using sql? I'd miss
all the view helpers etc available at present I think. Going from 30 servers
to 2 certainly sounds like a huge improvement, so it was definitely worth it
for you, are you thinking of writing any front-end apps in Go?

I've been playing around with Go recently and it is a fantastic language for
someone coming from dynamic languages like Ruby. I particularly liked
interfaces as a way to define a contract for implementations to follow, and
the simple package system which encourages you to make your code modular.

~~~
treeder
We actually have our own deployment tools for Go (and for Rails, our
databases, etc). We built them before all the new hip options that are around
today. We build on the target machines, although that's not a requirement
since we all run the same architecture on our dev machines too (64 bit linux).

I'm not sure about writing a front end in Go, there's not a lot in terms of UI
frameworks and Iron.io front end (HUD) is still in Rails so I can't really say
much about it.

~~~
grey-area
Interesting to know that you're building on target machines, I have yet to
explore cross compilation, but it'd be great to be able to build before
deploy.

Sorry, deployment was probably the wrong word - I was not so worried about
getting the files up there (like you I have a simple home-grown solution for
that), but more how you were handling process management, swapping out
processes etc.

Perhaps with Go this is less of an issue because startup time for new
processes is minimal, and you can simply kill one process, start another and
not really miss any requests (unlike say Rails with startup times of 10
seconds or so for instances)? Did you find this wasn't a big issue in practice
and something very simple works for you?

I'm currently playing with go but if considering it for client use would have
to be sure that things like this were not an issue. Are you using any off the
shelf tools for process management/load balancing or is it something that you
have built yourself? Did you run into any problems?

------
rurounijones
Ok, this bit has left me confused.

They were Java devs that liked ruby, they wrote applications in Ruby on Rails
and the ruby apps were hitting limits so they immediately started looking at
other languages.

But they don't mention the most obviously (to my mind) simple option.

JRuby

It is ruby (They like ruby). Most ruby apps can be run on JRuby with very very
little changes (No need for a big rewrite) and it runs on the JVM with which
they are familiar and it is very fast (and true multi-threading).

Maybe they did look into it but I would have thought that would be #1 on their
"We tried this but discounted it" list.

~~~
pkulak
In my experience, rewriting for JRuby is as much work as re-writing in a new
language. Last I looked was a year or two ago, so it may have all changed, but
it's not as simple as just moving your code over. Many really important gems
don't work, or don't work properly.

~~~
boundlessdreamz
The only gems that do not work are those which are not pure ruby. For the
popular gems I think there have been rewrites. I haven't looked into JRuby but
it has changed quite a bit in the last year from what I read

~~~
johnb
I don't think that's true. There are plenty of threadsafe gems with C
extensions, and plenty of non-threadsafe pure ruby gems that mutate
class/class-instance variable willy-nilly.

~~~
boundlessdreamz
gems with C extensions are not supported
<https://github.com/jruby/jruby/wiki/C-Extension-Alternatives>

I forgot about thread safety in pure ruby gems but that is not a huge problem
- [https://github.com/jruby/jruby/wiki/Gems-known-not-to-be-
thr...](https://github.com/jruby/jruby/wiki/Gems-known-not-to-be-threadsafe)

------
shurcooL
> "We also weren't sure if we would be able hire top talent if we chose Go,
> but we soon found out that we could get top talent because we chose Go."

I feel[1] that a smart/talented C/C++/anything developer can go from someone
who has never seen or heard of golang to a proficient and productive Go
developer in a matter of a few weeks, maybe even _days_, if not less.

That's how long it takes to go through the following materials (and fav some
for later reference) and play with the language a bit.

<http://golang.org/ref/spec>

<http://www.youtube.com/watch?v=ytEkHepK08c>

[http://commandcenter.blogspot.com.au/2012/06/less-is-
exponen...](http://commandcenter.blogspot.com.au/2012/06/less-is-
exponentially-more.html)

<http://tour.golang.org/>

<http://golang.org/doc/install>

<http://www.youtube.com/watch?v=XCsL89YtqCs>

<http://golang.org/doc/code.html>

<http://golang.org/doc/effective_go.html>

<http://golang.org/pkg/> \- use as reference

<http://www.youtube.com/watch?v=f6kdp27TYZs>

<http://talks.golang.org/2012/splash.article>

<https://gobyexample.com/>

<http://golang-examples.tumblr.com/>

And a some more similar things that you can mostly get to from golang.org
site. The beauty of how concise the language and even its website are, is that
you can literally just go through everything there one thing after another.

[1] This is my personal opinion based on playing with go the last few
weeks/months. I'd love to verify this theory. It's not yet the primary
language in which I do things in (I use C++11 atm), but for all my side
tasks[2] it proved to be indispensable. And I found it very easy to pick up. I
can't wait until I start doing all my work in Go, that will be a true test of
its productivity efficiency.

[2] <https://gist.github.com/shurcooL>

~~~
TillE
It's quite strange to me that people would identify as or look for a
"[language] programmer". Sure, I happen to write more C++, Python, and C than
anything else, but I've dabbled in just about everything and could reach
comfortable proficiency in a matter of weeks. Most of programming and all of
computer science is universal.

Any serious programmer should be a polyglot by default.

~~~
gilgoomesh
> Any serious programmer should be a polyglot by default.

It depends if you're going to spend years training someone or if you need an
expert right now.

My experience is that it is impossible to maintain expert level skills in more
than one or two language + library environments. You can remain familiar with
other environments but you don't have the time to be an expert.

While I sometimes switch between C-family languages for different projects, it
can take _months_ to get up-to-speed with the changes in a language and its
environment since you last programmed in it. I'm talking about situations
where I know the language well but the _environment_ has changed. Languages
change a little but the libraries they use can change _dramatically_. And
along with the change comes a whole body of implied knowledge about how to
safely and effectively use it all and this impacts on how you use the language
itself.

If you've literally never programmed in a language before, it can be a few
_years_ before you know about all the eccentricities, before you understand
why the language follows certain patterns, before you understand the risks
with certain behaviors.

If you need an expert now, not in 12 months time, then they need to know both
the language and the environment. If you can wait a couple months, then they
still need to know the language.

~~~
timr
_"It depends if you're going to spend years training someone or if you need an
expert right now."_

I'm not going to claim that there are no legitimate reasons to hire people who
are narrow experts in Blub (and only Blub), but I'm having a hard time
thinking of any.

At an established company, you've got the luxury of time -- there's rarely a
good reason to "need an expert right now" that isn't just a contractor. At a
startup, where hiring the wrong person is a disaster, hiring an "expert right
now" is like holding a loaded gun to your head. Ideally, you should be hiring
"T" people -- lots of breadth, with lots of depth in at least one area.

A truly good programmer will pick up your language/framework in days, and be
at full productivity in months, even from scratch. It's hard to do better than
that.

~~~
jbooth
Even at an established company, a really smart language newbie could write a
bunch of ad-hoc code that does almost the same thing as well-understood
libraries that everyone more experienced in the language uses. Even if his
code was relatively high-quality, now you have a bunch of extra stuff to
maintain, and everyone who interacts with it will have to learn this thing
instead of just using the library everyone knows. I've done this in languages
I'm unfamiliar with and will probably continue to do so.

~~~
timr
Not Invented Here syndrome goes away with general experience. It doesn't come
back every time you switch to a new language.

Just because I've never written Erlang doesn't mean that I will automatically
try to write a random-number generator (say) the first time I need one in
Erlang. I have enough experience to look for a library function first.

Empirically, NIH tends to be _more_ common in single-language developers, not
less. People who place a lot of value on their "expertise" in Blub tend to do
so because they're over-weighting the importance of their memory of the API
details. When they don't automatically remember something, they leap to the
conclusion that it doesn't exist. They're also typically a less-experienced
cohort than people who have written in lots of different languages.

~~~
jbooth
I wasn't talking about NIH syndrome, I was talking about "I don't know that a
common library exists for this standard use-case so I'm going to write my own
one-off because I have a job to do". I mean, you can google for libraries but
sometimes you just don't find them and then find out a few weeks later what
you should have used.

~~~
timr
_I don't know that a common library exists for this standard use-case so I'm
going to write my own one-off because I have a job to do._

That's called "laziness", and is even more likely than NIH syndrome to be
mitigated by experience. Certainly, having lots of coding time with a single
language doesn't make you less lazy, and having lots of experience with
different languages doesn't make you lazier.

~~~
jbooth
Well, I'm less lazy than most devs and have worked in lots of languages and
did it just the other week -- maybe I'm just not that bright :)

~~~
lifeisstillgood
Could you give a specific example as I am still thinking "random number
generator" and want to balance it with your "standard idiom or library that
everyone steeped in $lang knows "

Cheers

~~~
philh
I dunno how well-known the things I didn't know about are, but an example of
me doing it: <https://news.ycombinator.com/item?id=5113202>

~~~
lifeisstillgood
I think there is a fine line between using a library and writin your own to
understand better the domain

It's something to do with how critical the library functions are to you / your
system. I would never write my own compression software, but I can see why
people would just to learn about the trade offs.

~~~
philh
Okay, but if I had known about the alternatives I would have used them, so I'm
not sure how that's relevant.

~~~
lifeisstillgood
philosophising really. I guess on the "actually helpful advice" level, I'm all
out if the googles cant help.

------
phasevar
Excellent success story with Go.

Love this quote... "We also weren't sure if we would be able hire top talent
if we chose Go, but we soon found out that we could get top talent because we
chose Go."

~~~
GhotiFish
That's really obvious but overlooked. Developers just looking for their
paycheck will only have learnt the established languages.

Developers who enjoy their profession will look to the future, excitement
gleaming in their eyes.

I've heard of business developing in Haskell a good amount of success. I
suspect, for the same reason.

~~~
jmilkbal
It could also be said that developers who use established languages are
excited to get things done rather than reinvent wheels continuously.

~~~
thirsteh
No, it can't. People don't use only Java because they "just want to get things
done." They use only Java because that's all they learned, and all they need
for their day job. If not, then they will invariably step over languages that
are very interesting but obscure.

~~~
jmilkbal
I'll give an anecdote. I've spent the last 7 years using Ada, a language that
is /both/ interesting and obscure, almost exclusively. It's really a fantastic
language that is frequently neglected much like poor Go, but with more merit.
Anyway, in preparing the launch of my company's first iteration of its podcast
network (<http://76streetnetwork.com>), I absolutely wanted to use Ada, but
the allure of the utterly fantastic tool support and breadth of libraries
available for Java had me up and running in a day or two despite having never
written anymore than basic hello world-like crap with it in the past. There's
really nothing crazy about Java, you're right, but I live for coding and I
have a deep interest in quality software development, but I guess I'm just
looking for a paycheck for this side project that will never make me any
money! Ironically, Ada is what pays my bills.

~~~
thirsteh
Don't get me wrong, I'm not saying you shouldn't use Java--just that "I've
only ever used Java" is a pretty good indicator of somebody not being
genuinely interested in software development, and different ways to do things.
I'd take somebody who learned Ada for fun over somebody who learned Java in
school (and did nothing else) for a Java job any day.

------
riobard
“the entire process started up with only a few hundred KB's of memory (on
startup)”

There is no way this could be true: last time I checked, a bare minimal Go
HTTP server requires at least 2.8MB of memory on 64-bit machine. Are you using
the default net/http library?

~~~
codygman
Yeah, my HTTP server made in Go usually starts up at around 6MB or so.

------
rachelbythebay
This story sounds mighty familiar. I realize the topic is about one language
over another, and most of the comments here seem to be focusing on that, but
there's another part which I find fascinating: the cascading failure mode when
they'd lose a web head.

"This would in turn cause the load balancer to think it failed and take it out
of the pool, thereby applying the load that the unresponsive server would have
been handling to the remaining servers. And since the remaining servers are
now handling the load of the lost server plus the spike, inevitably a second
server would go down, the load balancer would take it out of the pool and so
on."

It was 8 years ago, and the customer was running PHP, not Ruby, but otherwise
it's the same basic story that lead to the creation of "Surge Protector", or
its actual name, "Suicide Pact". <http://rachelbythebay.com/w/2011/06/28/sp/>

~~~
pilsetnieks
Keeping in theme with the suicide pact, STONITH, the other kind of pact for
high-availability machines comes to mind - Shoot The Other Node In The Head.

------
rcb
Why did go "win" over erlang, scala, and clojure? Because it's becoming trendy
and "fun?"

And, why are "fun" and "joy" terms ruby bloggers use for languages? Is this
code for "easy" and "familiar?" As in, "this language is easy to learn.. It
does not require us to learn difficult but mind-expanding concepts to become
proficient."

Replacing ruby/rails with something (just about anything!) results in far
fewer servers. Isn't this obvious to all? Is this really news to the average
HN reader?

If that's an I/O bound API server, bet those two servers could be brought to
one in a language/runtime that's more productive but a little less "fun" to
learn.

~~~
rektide
One could adopt some kind of event-based system in scala or clojure, but
Erlang and Go are alike in being the lone runtimes where running millions and
millions of very small messaging processes is AOK no problem for the runtime,
and not something one has to work really hard for.

Fun and joy are terms applicable here because the alternative is Erlang. Zing!

Go is rather ideal for these guys use case: if it wasn't an easy win, given
what a snug fit their use case is, there would've had to have been red flags
abound, as this really is a nearly idealized work load for Go's use: a hell of
a lot of processes which sit around doing nothing, where one occasionally gets
a message and forwards it along. Perfect Go story, as Go's lightweight
processes (goroutines) are ideal for this kind of _Communicating Sequential
Processes_ routing workload.

~~~
Peaker
Haskell's runtime makes for cheaper threads than Erlang. Possibly cheaper than
Go's, too.

~~~
rektide
Are they also in Haskell as in Erlang pre-emptable by the runtime?

There's certainly some cost in this design decision, more context that has to
be swapped in and out and more state kept, but it's important capability to
allow blind design & use- think Node, where everyone has to keep re-iterating
how important it is to keep yielding to the event loop, to not do a lot of CPU
work in a handler: irrelevant in Erlang world- in spite of the threading model
not being OS threads, Erlang is happy to drop your lightweight thread on the
fly when it sees fit.

~~~
rdtsc
> irrelevant in Erlang world

Exactly. That is a subtle distinction but for cases where responsiveness and
low latency is important that is key. Another thing Erlang has is isolation of
process heaps. If one process crashes, it won't affect others. No shared data
structures between processes. It all goes to fault tolerance but also a major
win for a completely concurrent garbage collection.

~~~
masklinn
> If one process crashes, it won't affect others.

Unless they're linking or monitoring one another, an other important property
of tolerant and distributed system.

~~~
rdtsc
But that is on purpose. Sometimes you do want that. In other words they are
explicitly set up to monitor/link each other.

~~~
masklinn
> But that is on purpose.

Indeed, but I was pointing it out because it's important to be able to do it.

------
saosebastiao
This could easily be rephrased as "How We Went from 30 Servers to 2: Static
Typing and Compiled Code".

------
vitalie
I rewrote my DNS checking tool (<http://www.dnsinspect.com/>) in Go and I saw
huge differences in resource usage (previously it was implemented with Ruby on
Rails + EventMachine), my memory usage went down from 128MB per background
worker to a few KB.

Now I'm able to run hundreds to thousands of concurrent reports using a small
VPS, the Go application is using 36MB of RAM (24MB front end + 12MB for
background workers). Go language is well suited for my particular case (many
concurrent IO operations).

This was my first Go project, in a week I was comfortable with it, I had many
alternatives but I really liked the simplicity of Go, fast compilation and
easy of deployment. Because I've missed some pieces from the Ruby's world I've
combined Jekyll (Compass, HAML SASS, RedCarpet, etc) with Go. :)

~~~
swah
Do you also used the default net/http support libraries?

~~~
vitalie
Yes, I'm using net/http.

------
saidajigumi
I'm missing the connection between poor API performance under Rails and the
decision to do a ground-up rewrite. Was the initial performance problem due to
an API problem or a platform problem? Was there any profiling of the poorly
performing API server? I'd love to hear the story behind that analysis and how
it impacted the tale told in TFA.

I ask all this since I've lost count of how many times I've participated in or
witnessed an averted rewrite via a good dose of profiling and a few key
bugfixes. I'll acknowledge that's not as fun as a clean-slate project, but
_vast_ amounts of engineering time were saved.

------
conorh
I agree with the 'Go is fun' (and easy to pick up) sentiment. I spent a few
weeks recently writing an NTLM library for Go for a consulting client (I hope
they'll let us open source it at some point). All the parsing and bit
manipulation was _extremely_ straightforward due to the excellent standard
libraries and I was able to quickly move from not knowing Go, to a fully
implemented NTLMv1/v2 library!

------
pjmlp
Another day, another Go PR post on HN.

What is new about porting code running in one of the slowest interpreters
around to a compiled implementation of another language?

I was already porting Perl/TCL code to C++ back in 2002 with similar
performance results.

Don't kids nowadays learn anything about performance in their CS degrees?

------
zenocon
It's an interesting tale, and I honestly have no bias toward Ruby or Go, but
the cause/effect relationship is poorly illustrated (we switched languages).
We all know runtime / performance / scalability is more complex than that. I'd
like to hear where the bottleneck(s) were and how Go solved them.

------
donebizkit
Last month I picked NodeJS and build couple of sites with it. This month, I
want to pick up something new. I was hesitating between Python and Go. Can you
answer couple of questions about Go from someone coming from NodeJS:

1- What is the state of the external Go libraries, especially DB (MySql),
caching libraries (memcached), protocol libraries (Oauth). Are they stable
100%

2- How easy is logging and tracing in go?

Thanks.

~~~
paddyforan
Memcached's author (Brad Fitzpatrick) is a Go user, and wrote a memcache
client for Go.

Go has a bunch of good stuff surrounding SQL. I haven't used MySQL,
personally, so I can't comment on which MySQL driver is best, but I know
YouTube uses Go for something related to MySQL, though I can't say for certain
exactly what. <http://code.google.com/p/vitess>

The goauth2 (<http://code.google.com/p/goauth2>) library is written by Brad
Fitzpatrick and Andrew Gerrand (two members of the Go team), and is stable.

Logging is great. Interfaces make it really, really flexible. I'm not sure
what you mean by tracing. Stacktraces? Those are easy to retrieve:
<http://golang.org/pkg/runtime/debug/#Stack>

~~~
tptacek
Well, Go has a bunch of libraries for different SQL servers. I wouldn't go so
far as to say it has lots of good SQL stuff; it's in approximately the same
place as C is w.r.t. databases.

Having to interface with complicated SQL is a reason _not_ to use Go (but not
an insurmountable one).

~~~
burntsushi
> it's in approximately the same place as C is w.r.t. databases.

I think the existence of a standardized interface like `database/sql` makes
things strictly better.

An anecdote: I started making a new web application using MySQL. As soon as I
realized the error of my ways (not because of the driver), I swapped out a
single import from the MySQL driver to the PostgreSQL driver, and everything
continued to work as it did before.

[EDIT] Anyone care to explain the down-votes?

~~~
redler
The downvotes are likely because they're reading your MySQL-to-Postgres
anecdote as the firing of a shot in an unrelated battle many love to hate, if
not a mild troll.

~~~
burntsushi
Ah, right. My bad. The folly is still fresh, and there is still a bit of pent-
up rage that I hadn't made this discovery sooner.

------
laumars
Has anyone had an experience of running websites from Go; or more
specifically, how you handle none-HTML content?

I've been considering porting my CMS from mod_perl to Go, but I'm not sure how
you work with the other files (CSS et al). I did read somewhere that you run
Go from Apache but it's not recommended.

~~~
carbocation
I've just started creating toy projects in Go. I put these Go projects (and
Apache, which runs older PHP projects) behind nginx.

For hosting CSS, you either generate it programmatically and send the right
Header, or you can use http.FileServe from the standard library [1]. (Surely
other approaches are possible, but those are the two I've played with so far.)

[1] <https://code.google.com/p/go-wiki/wiki/HttpStaticFiles>

~~~
laumars
Thanks for that.

The issue I have is that I would prefer to keep Apache around if I can (I have
a few sites - some of which I host for friends). While my web server does have
a few IPs attached, I'd rather not have to buy more IPs just to separate
Apache from Go.

And to be perfectly honest, I do quite like Apache. (each to their own I know,
but I've had little reason to complain about it).

~~~
burntsushi
I use Apache too (it's all I've ever known), and run a few Go web apps.
Everyone lives quite peacefully together. With Apache, I just set up a simple
reverse proxy.

Here's an example of one I use to run my own `godoc` server:

    
    
        <VirtualHost *:80>
            ServerAdmin admin@burntsushi.net
            ServerName godoc.burntsushi.net
            ProxyPreserveHost On
            ProxyPass / http://burntsushi.net:8080/
            ProxyPassReverse / http://burntsushi.net:8080/
        </VirtualHost>

~~~
laumars
Nice idea.

I might try that before dipping my toes into Nginx

~~~
burntsushi
Yeah, it's a nice holdover. I keep hearing great things about Nginx though.
Someday I'll have time to check it out :-)

------
neya
It's funny that they argue that they dumped the JVM derivatives like Scala,
but wouldn't exactly tell why, nor support their argument with any sort of
data/logs.

I am genuinely curious to know why they chose Go over Scala. If it was the
syntax, etc. I can partially agree because it's one of scala's weak points,
but then they pitch the main reason citing performance, so I'm genuinely
curious to know.

~~~
fauigerzigerk
He mentions JVM memory usage.

~~~
neya
I think he edited it after me and many other posted similar comments, coz I
didn't notice it the first time...

------
TakeTwo
"We decided to rewrite the API" and now it is over 300x faster. Go had nothing
to do with this, it was obviously some boneheaded original design. Probably
something like fork+exec on a Ruby VM on each request to enter a chroot.

I know HN has a hard-on for Go but come on.

~~~
drivebyacct2
Are you the same asshole making throwaways to shit on all of the Go postings
or are there actually more than one of you going around making assumptions,
accusations and spreading falsehoods about Go.

I can't get over how pathetic it is to make a throwaway for that comment, I'm
more embarrassed five people validated your comment and attitude.

------
topbanana
Alternatively...

why we needed 30 servers: Rails

------
siavosh
I've read this domino-effect on server clusters on several HN postmortems and
I've seen various flavors of this on our own web servers. I'd like to think
there's a simple configuration in most servers that prevent 100% cpu
utilization from taking place and preventing the server from telling its
cluster that it's still alive. Anyone have any experience with this?

~~~
laumars
It's not really possible without adding resources (ie a new server).

The problem is, once your servers get saturated, requests queue up, processing
slows down and users start to hit refresh which artificially and exponentially
increases traffic. Then to compound things, servers buckle, which means you
lose a resource right when you needed it the most.

A domino effect is one analogy, another might be how a small hole in a damn
can buckle and blow open from the force of the water pushing through.

~~~
ef4
> It's not really possible without adding resources (ie a new server).

That's not quite true. With the right architecture, performance will degrade
gracefully. You'll get slower and slower responses as the load goes up, and
eventually start dropping requests, but the servers will not die and there
will be no cascading failure.

One way to achieve this is to make sure the queuing happens at the load
balancer, and no large queues are allowed to build up in the individual
application servers.

~~~
laumars
Well yeah. You can do all sorts of tricks from TCP/IP hacks to streamline HTTP
requests through to disabling queuing entirely. But my point is you cannot
entirely prevent your site from saturation without adding extra servers to
your web farm (and more so, that _siavosh_ 's method of taking servers out of
service has the inverse effect of what he was trying to achieve).

Thus all you can do is slow the escalation in the hope that the traffic peaks
before your resources buckle.

The only method I'd found that is "guaranteed" to prevent such outages is the
use of sorry pages (ie a static page stating "We're experiencing high volumes"
which users are directed too if the dynamic page connections are maxed out).
However even that is just essentially a prettier version of a page time out -
and I mean this in terms of usability rather than technicality. ie the site is
still unavailable, but you're killing the connection in a user friendly way
rather than allowing connections to stack or just flat out disallowing "> _n_
" active TCP/IP connections.

------
jscottmiller
Can you describe your workload a bit as well as any benchmarking you did to
determine (and perhaps optimize) hotspots in the ruby code before you started
the port?

Personally, I've been interested in moving to Go on a python-based project of
mine. Thus far, I've avoided it because 1) the extra work required to self-
implement a few third party libs I rely on and 2) I've been able to eek out
sufficient performance using c extensions and cython.

~~~
treeder
This first rewrite in Go was for the IronWorker API so all the operations are
here: <http://dev.iron.io/worker/reference/api/>

There were less endpoints back then, but you can get the idea. The most
heavily used operation being queuing up tasks/jobs.

------
dschiptsov
Science and engineering from Bell Labs behind Go is what make this possible.
It is not a "different language", it is carefully selected design decisions
and ideas behind it.

It relies on a principle of being good-enough, not to stuff everything in, as
a "feature sellers" and "buzzword shouters" used to do.

There are also lot of work of great minds behind Lisps or Erlang, and same
principles in a foundation.

------
ajanuary
"So then the decision came down to which language to use."

I find this tendency to constantly jackknife between talking about languages
and talking about frameworks a little dizzying.

Shouldn't the decision have been what web stack to use? I'm sure you could get
half way there with a bespoke Ruby web stack, maybe built on top of JRuby or
with a sprinkling of C extensions. You could get the same poor scalability
with some badly designed framework on top of Go [1].

There's a lack of meat in the post (hence why people are calling it out as
PR). How do these features of Go make it easier to write applications or
frameworks that cater to this particular scalability need? Could extra effort
be put in up front to get something similar out of Ruby or any of the other
languages, or will they always be sub-par? What is it about Go that makes it
light up the rest of the stack in a way other languages don't?

Or have you just traded one trendy technology for another because it promises
to be the magic bullet for your current itch?

[1] This isn't saying Rails is badly designed.

------
jianshen
Hey that's my photo of the GOpher! :)

[1]
[http://www.flickr.com/photos/jianshen/8080852738/in/photostr...](http://www.flickr.com/photos/jianshen/8080852738/in/photostream)

~~~
treeder
Hah, nice photo! Are you ok with that? Found it on google images.

~~~
jianshen
It's cool. :)

------
johncarpinelli
OK, I am sold on Go as my next language for web applications. Unfortunately, I
can't throw away the existing PHP/MySQL code. Any suggestions on how to
integrate Go and PHP into a single web server? Should I be using FastCGI for
Apache to call the Go programs?

------
jabagonuts
Out of curiousity, I would be interested to know which version/patches of ruby
you were using. With ruby I'm used to running out of memory on servers long
before running out of CPU. Of course, most of our workloads are IO bound, so
that may be the diffent.

~~~
treeder
This was on Ruby 1.9, 2010 era. No patches. Memory was definitely a concern
and probably contributed to some of it. Just to note though, after a machine
was taken offline by the load balancer, it would generally come back to life.

~~~
TresAmiga
Memory "probably contributed to it"? You don't even know _why_ the old version
was slow but you want to credit golang with making it fast? Hype much? The
lack of critical thinking here on Hacker News is astounding.

------
orangethirty
What changed in the architecture? It is rather interesting that by just having
a language change you could remove 28 servers from the system.

~~~
ikeepforgetting
My guess is that removing 2-5 layers of abstraction can get you pretty far
along.

~~~
buro9
We're writing most of the systems for our company in Go, and of the many
reasons this ranked highly. Just removing the multiple unnecessary
abstractions gets you a hell of a lot of simplicity and performance back.

~~~
ikeepforgetting
I work in an area that requires a lot of high performance computing and there
is a reason people in this area like code simplicity. Stacking layer upon
layer of complex algorithms is a lot more difficult to reason about than
having a simple and elegant system that's close to the metal.

There is a recent trend of building more and more complex systems, have less
coupling and 'good oop'. That's all good and nice but simple code is:

    
    
      - more maintainable (a complex nest of objects & function calls all loosely coupled isn't)
      - a lot more readable
      - easier to optimize
      - better for your sanity
    

Some languages have build a culture of extreme complexity and are notable for
building huge structures and systems to do simple tasks. Having a 50K lines of
code dedicated to just inversion of control or O/R mapping makes a problem a
lot more complex than it probably needs to be.

~~~
martinced
_"Having a 50K lines of code dedicated to just inversion of control or O/R
mapping makes a problem a lot more complex than it probably needs to be."_

Exactly. That's the terrible stuff with all this "enterprisey" mindset: people
are working on medium-sized codebase made of this special kind of hell that
Java/C# + ORM ([N]hibernate) + XML + SQL is and these app often run into the
200K / 300 KLOC lines if not more. Yet what does these applications _really_
do? Actually not very much. Yet these programmers are sure to be working on
super-advanced stuff because their _codebase_ is big.

When several companies reported a drop in LOC of 90% by switching to something
else than Java then at one point you have to at least consider that maybe most
of your Java codebase is hot air.

------
goatslacker
It would be interesting to hear some of the reasons you went with Go vs
Clojure (and also the other languages you mentioned)

------
ColinWright
I know I'm late to the party, and no one will read this, but ...

    
    
        "... Java derivatives like Scala and Clojure ..."
    

Someone help me out here: In what way is Clojure a Javav derivative?

~~~
willvarfar
Perhaps in the sense that they are languages that target the JVM and that rely
on interoperability with the Java standard library to bootstrap people making
apps with them?

~~~
ColinWright
I can't see how it could mean anything else, but I find that, well, bizarre.
Oh well, different point of view, I guess.

------
taproot
> with Ruby.

Title should have been: "How we went from 30 servers to 2: Replacing Ruby"

------
rmoriz
How about BDD, test-driven development and quality insurance? Does Go provide
an ecosystem that supports agile refactorings that are common in lean
startups?

You can say anything bad about the performance of Ruby and Rails etc. but
rspec, cucumber, capybara, vcr, factorygirl are really important features to
start from zero and reach a viable product.

~~~
trungonnews
Go doesn't have any mocking framework like Mockito/Java or Rspec mock. You
will end up writing a ton of code yourself if you want to mock something
simple.

~~~
drivebyacct2
Uh what? No, that couldn't possibly be less true. Have you ever heard of
"interfaces" in Go? They're kind of an important feature.

~~~
trungonnews
That's what the other gophers say too. Java has interfaces too, but it also
has Mockito that makes testing much easier. You should look into it. Mocking
library will help reduce boiler code. You would know that if you had used one.

~~~
drivebyacct2
>You would know that if you had used one.

Cute.

~~~
trungonnews
My bad. :)

------
free652
Uh I think I love it. Easy to learn language and awesome performance.

------
programminggeek
I wonder how far they could have got going more of a bare-bones Sinatra app
using JRuby. It's still not nearly as fast as Go obviously, but take away
Rails and some of the more dynamic usages of Ruby on a faster VM and I'd
venture to guess you might have been able to go from 30 servers to 10-15?

That's still not 2 mind you...

------
fernandezpablo
There's simply no technology that makes you switch from 30 machines to 1 (you
say you keep 2 just for reliance).

Obviously there must be other architectural changes in play here (I'm guessing
RoRs thread-per-request vs some kind of event loop on Go?). Please be more
specific.

~~~
genwin
Go is good at keeping all the cores on the machine pumping. Maybe they
switched to machines with more cores.

------
s_baby
Why Go over VisualBasic?

------
moron4hire
There are a lot of things that can be said about the "average" programmer in
such-and-such languages, what they do and don't do, what habits they have,
etc. But I'm not interested in working with merely average people.

~~~
smrtinsert
That's right. People like you and me work with the top 5% of programmers! It's
because I'm smrt.

------
dubcanada
You have an error in Opera

'SyntaxHighlighter.config.bloggerMode = true;'

Undefined variable: SyntaxHighlighterError

------
knodi
Go 1.1RC just around the corner in April. A lot of good stuff in Go 1.1RC.

------
chmike
If you look for a Golang programmer today, it's more likely you'll find a
_true programmer_ instead of a _me too_ programmer. So if you're a true
programmer, learn Golang. ;)

------
S_A_P
Now I know that the dynamic languages such as ruby and python are not known
for performance, but the benchmarks I have seen with Go look like it isnt
exactly the fastest language available either. After reading this article I am
more apt to think that something is/was VERY wrong with their Ruby code than
to think that Go is that much better. Maybe my lack of experience with either
GO or Ruby is showing here, but Ruby cant be _that_ slow can it????

------
rektide
p.s. come disrupt our business: with nearly no hardware, no costs, and no
maintence burden you too can pass OMG huge amounts of message traffic.

this is one bold as fuck post.

------
jebblue
<http://www.dartlang.org/support/faq.html>

>> Q. How does Dart relate to Go?# Dart and Go are both language projects
started at Google, but they are independent and have different goals. As a
result, they make different choices, and the languages have very different
natures, even while we all try to learn from each others' work.

Hmm, don't think I want to bet the farm on that.

~~~
drivebyacct2
Don't want to bet the farm on what, do you mean Dart or Go sticking around? Go
is used extensively through Google. It powers a significant DB layer that
powers YouTube. It serves up all of Google's static downloads and several
employees hint at other significant projects they can't speak about.

I've only dabbled in Dart, I can't speak to how vital it is, currently it only
seems viable when translated to JS or when using in a server-side VM, which is
certainly an option if you want.

(sorry, I originally had written Google where I meant YouTube, see "Vitess"
for more info)

------
st3fan
_We decided to rewrite the API. This was an easy decision, clearly our Ruby on
Rails API wasn't going to scale well and coming from many years of Java
development and having written a bunch of things that handled tons of load
with way less resources than what this Ruby on Rails setup could handle, I
knew we could do a lot better_

------
Tzunamitom
Was anyone else expecting this to be a boast from EA about how they cut their
operating costs for the SimCity launch?!?

------
mrwnmonm
man, you just made me worry very much now - we are making taxi dispatching api
using ROR - and we will lunch it soon

~~~
thibaut_barrere
Why worry? If you can handle the initial load, you can still optimize later
on, rewriting some functions or parts of the system with something faster.

~~~
mrwnmonm
great

------
trungonnews
I wish Go has named arguments like Python and Scala. Reading and debugging
code could be so much simpler.

------
petrohi
Why Go over Erlang?

~~~
Ixiaus
(Note: I'm a big Erlang supporter and 90% of my startup's code is written in
Erlang)

Erlang is _not_ memory efficient (compared to Go? I don't have benchmarks to
back this up) AND the learning curve for developers not exposed to functional
programming (which is a lot) is quite a bit higher than it is in Go.

------
seivan
Hmm, wouldn't it be better to rewrite the ruby services to Go, the actual
workers and not the entire API? I don't think the API HTTP requests where your
main issues here...

------
jroseattle
Would be great to hear more specifics of the rewrite. What did the API do,
what pieces were built in RR, what got replaced in Go, etc.

At least as much as you're willing to share. :-)

------
seivan
The problem was Ruby, not Rails. Did you give ever give Jruby a chance?

------
fwee
Rails is not for API,why not use eventmachine?

------
bokglobule
Why was Go chosen over NodeJS ?

~~~
treeder
Mostly because I can't stand Javascript. It makes me cringe just thinking
about it. Go is much nicer to work with.

~~~
methehack
Hey OP! I appreciate your sharing. Since you came from ruby and we're on the
topic of the language itself, I'd appreciate your impression of how well Go
supports collections. Is there or could one write something like
<http://underscorejs.org/>? Can you do this kind of thing?

[1, 2, 3, 4, 5].reject {|i| i < 3}.map {|i| i + 9}

I did the go tutorial the other day and I became a little worried that one
would not be able to do this kind of thing without a bunch of unwieldy
declarations. I think that would be a showstopper for me.

Thanks for any insight you might have!

~~~
jbooth
Go has first-class functions, so you could write a library do the same thing
with, yes, a few more declarations around it. In your example, .map { |i| i +
9 } is .map( func(i int) int { return i + 9 }), or
.map(myFuncDefinedElsewhere) in most real-world scenarios.

As a commenter upthread pointed out, Go and serverside Javascript are aimed at
very different use cases and audiences.

~~~
paddyforan
I thought this would be a fun exercise, so I implemented the Go equivalent.
Can anyone get it to fewer lines? It abandoned all pretenses of readability
long ago.

<http://play.golang.org/p/M5VaeIgQ-q>

~~~
paddyforan
Updated to generalise a bit more. If this ever came up in code review, I'd
probably facepalm pretty hard.

<http://play.golang.org/p/YUQSZgFx_b>

~~~
jbooth
Ha. Yeah, the map() and reject() function defs are fine but the actual usage
is a little eye-bleed inducing :)

~~~
paddyforan
So now that we've established Ruby is better at being Ruby than Go is, I
wonder if there's an actual use case for this kind of thing, so I can
demonstrate the "Go way" of handling it...

------
ksec
No Love for Lua and LuaJit.

------
drivebyacct2
Man, what is it about Go that just brings out the pricks of HN?

There are over three throwaway accounts created exclusively for this thread to
shit-post about Go, to steal an old term I have never, ever used on HN before.

~~~
jff
I'm not exactly sure, I've noticed the same thing.

Go appeared a few years ago, quickly became pretty well-known, and continues
to grow in popularity. Maybe it's seen as "threatening" to older projects like
Ruby and Python? The most vitriolic comments always seem to come from Ruby,
Python, and Rust partisans. Of course, I'm not sure why they're so threatened
by it, they themselves explain in great depth how the lack of generics or
exceptions will prevent anyone from writing any real code in it, or even
trying to in the first place!

------
slem311
great stuff!!

