

Stripe open-sources Einhorn - gdb
https://stripe.com/blog/meet-einhorn

======
tptacek
This looks extremely useful. We have several EventMachine proxies that I'd
love to make redundant and load-balanced with something like this.

I took a couple minutes to try to get one of them "einhorned"; I lost most of
my time to trying to satisfy EM gem dependencies (we use em-http-request,
which has a fussy gemspec). I got Stripe's EM built, file descriptors passed
to my proxy, and EventMachine::start_server set up with the fd instead of the
address; I'm getting accept callbacks, but not received data.

Will tinker later. High hopes. Thank you very much for open sourcing this.

~~~
gdb
No problem! Feel free to ping me (gdb@stripe.com) if you run into issues.

~~~
Bjoern
Are you hanging around in the IRC somewhere?

~~~
gdb
I lurk around #stripe on freenode usually.

------
cperciva
Depending on your platform, how many children you've forked, and your rate of
incoming connections, this design could expose you to a thundering herd.

You might find that you get better performance by having the parent call
accept() and hand off sockets to the children. With sufficient abuse of
LD_PRELOAD you can probably do this without making any changes to the child
processes -- you just need to override accept() with a function which checks
if any sockets have been handed off.

~~~
gdb
That's a fair point. Even if your accept() implementation is fine, event-
driven workers would instead be select()ing on the shared socket, which can
also lead to a thundering herd. I think you'd need quite a few worker threads
before this became an issue in practice though.

~~~
cperciva
Indeed, it's an edge case... but it's a painful one if people do run into it.

------
coderdude
(The following is off-topic:)

I noticed that the Stripe blog doesn't allow comments. I think most people
find it odd or frustrating when they can't add their 2 cents to something (for
better or for worse!). Any idea which things might have driven this decision?

\- Not wanting to deal with how most people act on the net? (Or maybe they're
in a highly competitive space where the juggernauts wouldn't think twice about
trying to destroy their image.)

\- To encourage people who want to discuss the blog entry to submit it to
their favorite discussion communities?

\- To avoid possible PR issues sitting right on their company's blog? (Related
to point #1)

There is a kind of beautiful simplicity to the page that comments would
probably destroy.

~~~
justinsb
I made the same decision on my personal blog. When there are comments both on
a blog and here on HN, the conversation feels unnaturally fragmented to me.

Further, when I did have both, comments that self-selected the blog (vs HN)
are - in my opinion - much more likely to be spam/promotional, or generally
less 'valuable' (e.g. more likely to be simple questions that are already
answered in the post).

~~~
coderdude
There's an angle I hadn't considered. The spammy and promotional type of
comments would be far more likely to appear directly on the blog since that's
where the value (for the commenter) is in leaving them. That's something that
moderated comments would control, but you can't moderate your way out of
fragmentation.

I would expect more negative comments to be left directly on the blog as well
since the commenter won't have to deal with the potential backlash from their
community. Which is probably also a good reason to never allow any kind of
anonymous of guest comments. (Although that wouldn't stop the truly motivated
negative commenters.)

~~~
Danieru
Moderation can also get tricky. Pure spam is simple but you will run into
comments where the commenter somehow works in their (product || experience).
So long as they provided some commentary it is not pure spam. So the situation
becomes a bit touchy.

This is a problem we ran into at PhpPhotoUploadr. In our case we have a very
talented team so we hacked together a solution.

PS: We're always hiring 忍者!

------
waffle_ss
Am I to understand that Einhorn doesn't require the use of Unix sockets to
delegate work to workers? If so, it could work quite nicely for JRuby apps as
Java doesn't support Unix sockets and it's therefore not possible to just plop
nginx in front of the Web server (Tomcat or Jetty) instances and have it queue
up requests to a shared Unix socket like the typical Unicorn setup does.

~~~
gdb
That's correct. In fact, at the moment Einhorn only supports TCP sockets.
(That being said, I might be mistaken, but I think that Unicorn can also bind
TCP sockets.)

~~~
waffle_ss
Thank you for the follow-up. Even though Unicorn does support TCP sockets, it
is irrelevant for our JRuby apps as I believe (IIRC) it won't even install
under JRuby due to being written in C only, with no native-Java or pure-Ruby
alternatives available. So it's not a Web server that can be used for serving
JRuby apps.

Now, since you've extracted the capabilities of Unicorn out into something
that isn't coupled to a Web server (and it's written in pure Ruby, so is
portable), we should be able to use it as a shim between nginx for managing
Trinidad (Tomcat) instances. Very much looking forward to giving it a try -
thanks for your efforts!

EDIT: Whoops, guess I spoke a bit too soon. I get this error:

    
    
      NotImplementedError: fork is not available on this platform
    

I think a typical workaround on JRuby is to use the posix-spawn or spoon gems.
If I have some time this next weekend maybe I could test this a bit more and
maybe formulate a patch. Example posix-spawn usage [here][1].

[1]:
[https://github.com/ddollar/foreman/blob/master/lib/foreman/p...](https://github.com/ddollar/foreman/blob/master/lib/foreman/process.rb#L54)

~~~
gdb
Ah, interesting. Yeah, would love a patch!

------
jefe78
Awesome news. Still waiting for you guys to come to Canada so I can give you
all my money!

~~~
gringomorcego
is stripe only available to us customers trying to buy through your site? or
only us startups, and all of their customers?

~~~
cperciva
Only US merchants. Customers can be anywhere (except North Korea, Iran, etc.
presumably).

------
mhd
The logo is confusing me a bit. What's the connection between icosahedral gems
and unicorns? Is this some pop culture reference I'm missing or is it
basically a stopgap considering that Unicorn itself already has an eponymous
logo?

~~~
gdb
Yeah, it's mostly a stopgap.

------
chadrs
Does this work for long-lived connections? Say I've got 1000 TCP connections
open to my backend workers and I want to roll out new workers. Will the old
connections die or survive the config change?

~~~
gdb
Yep, it will indeed work. New workers will be spun up, the old workers will
get a signal asking for a graceful restart, and those old workers can hang
around for as long as they need. You'll have twice as many workers alive as
you might like, but there's not much Einhorn can do about that.

------
mrinterweb
I'm interested to see some more posts about Einhorn. I would like to see some
example use cases for it. Looks like a very interesting project.

------
soperj
i'm going to fork Einhorn and call it Finkle.

~~~
justinjlynn
Finkle and Einhorn... Einhorn and Finkle... Finkle is Einhorn... Einhorn is
Finkle...

~~~
greghinch
Einhorn is a man!

------
aneth2
I just heard of Unicorn / Rainbow as a way to get more out of Heroku
workers/dynos. I wonder if this is also compatible, or a better alternative?

I'm new to this whole concept, so am just about to get my feet wet.

Hopefully, Heroku will implement something like this internally, as it seems
it would help them to eliminate the greatest flaw with their system - that
each deployment require 5-10 seconds of downtime as new dynos are started.

------
altrego99
To my knowledge both FastCGI and Unicorn are not free. This is definitely an
excellent alternative to those - and probably even easier to use for some use-
cases. I am glad and thankful that Stripe decided to make it open-source.

~~~
gdb
Actually, Unicorn is indeed free (you can grab the code from
git://bogomips.org/unicorn.git -- looks like it's GPL). FastCGI is an open
specification, with many free implementations. There's definitely a lot of
cool prior art in this space.

