

How Grooveshark Uses Gearman - wanderr
http://wanderr.com/jay/how-grooveshark-uses-gearman/2011/03/27/

======
encoderer
I implemented Gearman at a previous startup and then recently have done a
"clean room" re-implementation of it for a personal project.

First: Gearman is foremost a job quque, as others have mentioned. Obviously
most jobs have SOME data submitted along with them.

Second: After implementing it as a job queue, another team down the road
needed a message queue. So we ran benchmarks. Turns out, Gearman was a faster
Message Queue than the dedicated *MQ platforms. We were getting killer
performance out of the Gearman Java client. Of course, we had to spend 2 weeks
fixing bugs on the Java client to get it to work.

Which brings me to my last comment... Gearman is NOT a mature system. It's
brilliantly fast and very good at what it does, when it all works. But you may
find yourself, like I did, getting personal with the Gearman code itself to
explain wonky behavior. Like, hey, why the heck isn't this job timeout being
enforced? Fail.

But as for being awesome, let me give you a taste:

It has a MySQL client. So on our reporting database servers, we installed the
gearman client. Which meant we could create UDFs in MySQL that would be
actually backed-out by Gearman. In other words... we could do something like
this:

Select user_id, UserRiskScore(user_id) from users where user_id = 123;

And UserRiskScore kicks off a Gearman job... which in this case could be just
an entry point for a Gearman map-reduce: it dispatches many other jobs,
aggregates the result, and then returns it to MySQL and your waiting query.

And, of course, the actual worker code can be written in languages far more
useful than SQL.

Now, I'd never install a system like that on the live database. You don't need
200 queries waiting on a gearman response when you've got buckets of data
innodb is trying to flush to disk.

But on our reporting db cluster it was brilliant. It made our code wonderfully
centralized. A FRONT-END coder can get complex results, having kicked off a
map-reduce job without ever having to know what, say, HDFS, or MongoDB, really
is.

------
wildmXranat
We have a similar setup for our application I have to say that I agree with
the article. Some of our workers work in tandem to solve small data chunks,
some do async fire and forget types of jobs. All in all, gearman was a
pleasure to work with.

As far as the memory job queue not being reclaimed, that might be mitigated by
using a secondary storage for it ala mysql via libdrizzle.

------
latch
I'm curious as to why Gearman vs something else, say RabbitMQ?

~~~
j2d2j2d2
Or zeromq?

~~~
wanderr
I don't think 0MQ existed when we made the decision to go with Gearman, but a
30-second glance tells me it would probably be well suited to build a job
server on top of, but isn't exactly an out-of-the-box job server like gearman
is. For one thing it's not immediately clear to me how you would submit jobs
via zeromq for which workers don't exist yet or aren't currently connected.
That's just a first impression though since I haven't dived into it very
deeply, obviously.

~~~
j2d2j2d2
I didn't mean to knock the decision about choosing Gearman. I was just
interested in your thoughts on comparing the two.

That's correct that ZeroMQ isn't a server. I tend to describe it as a sockets
layer that has messaging semantics. If a host isn't up, you can still send
messages to the socket. It queues them up, like you'd expect a messaging
system to do.

So this also means you can open a socket to a host that doesn't exist. Imagine
you turn a server on with no clients up yet. A PUSH socket will block. When
you connect a client, by opening a PULL socket, the message exchange starts.

These socket types can be inproc, ipc, tcp or pgm. inproc is an in process
queue. ipc is basically unix sockets. tcp is... and pgm is pragmatic general
multicast.

To build a broker, you'd probably use something other than PUSH/PULL sockets.
You would use REQ/REP (request/reply) sockets. These sockets give you
synchronous communication. The request sends a request and is guaranteed a
response.

I think their doc explains the rest better than I could, I recommend reading
this: <http://zguide.zeromq.org/page:all>

I think this description offers the gist of how REQ/REP, and their nonblocking
counterparts XREQ/XREP, can be used to basically script together your ideal
broker for many conditions.

 _When you use REQ to talk to REP you get a strictly synchronous request-reply
dialog. The client sends a request, the service reads the request and sends a
reply. The client then reads the reply. If either the client or the service
try to do anything else (e.g. sending two requests in a row without waiting
for a response) they will get an error.

But our broker has to be non-blocking. Obviously we can use zmq_poll(3) to
wait for activity on either socket, but we can't use REP and REQ.

Luckily there are non-blocking versions of these two sockets, called XREQ and
XREP. These "extended request/reply" sockets let you extend request-reply
across intermediate nodes, such as our message queuing broker._

