

Ruby and Elixir: Polyglottin' FTW - linc01n
http://www.neo.com/2014/08/11/ruby-elixir-polyglottin-ftw

======
rubiquity
I've done a setup similar to this in the past and really enjoy it. If you ever
find yourself writing logic in your Resque/Sidekiq jobs that has to do with
manually managing a data structure in Redis with regard to your jobs, your app
is screaming for a language with better tools for modeling complex workflows.

A lot of people are hitting on the performance improvements (25 Sidekiq
threads/workers versus 10k-100k+ Elixir/Erlang processes) but when I find
myself doing this it's actually because my background process flows are really
complex. The drastic increase in concurrency is just the cherry on top.

Background work is asynchronous by nature and often times you have
dependencies between jobs, groups/batches jobs and also need robust error
handling/recovery. This is where Elixir/Erlang really shines and using Actors
& OTP to model these complex flows is the huge win in my experience. Sidekiq
Pro offers support for batches which covers some of this, but ultimately
Elixir/Erlang processes are just a huge improvement in modeling such systems.

A concrete example is sending off 10 jobs. When all 10 are done, move on to
the next step. Each of these 10 jobs can also send data to the waiting step
which it can then use as arguments. If any job should fail N times, cancel and
rollback all of the jobs and try again. After the next step, split those 10
jobs into two groups of 5 and do those 5 sequentially. So on and so forth. Try
doing this in Ruby. You'll likely have to manage your own data structures in
Redis. In Elixir you just use GenServers and sometimes ets tables. It's all
handled in the language itself, rather than external data stores.

Sidekiq is probably suitable for 90% of Ruby apps doing basic fire and forget
and batching with Sidekiq Pro. But if your domain has seriously complicated
background processing, use a tool more suited for that.

------
themartorana
Best thing I ever learned - probably 15 years ago at this point - is just use
the right tool for the job.

We're a mobile game company, and we use C, C#, Objective C, Java, Python, Go,
Erlang, JavaScript, Lua, and more where they make sense.

I love language talk and debates (and hate the language hate) here on HN. But
if you're only using a single language in production, you're likely not that
big, or more likely have a lot to gain by reworking certain bits with other
technologies where they make sense.

~~~
pjmlp
I never got those "I am language X developer".

It is like "I only do hammers and nails. For drilling ask the guy over there."

~~~
dagw
On the other hand if I want some intricate decorative wood carving, I'm going
to seek out a guy who specializes in intricate decorative wood carving, not a
guy who says "well I've never done any carving, but I've sawed, drilled and
hammered lot of wood in my day and carving is basically the same thing."

~~~
pjmlp
From my point of view you are mixing tools with domains.

Decorative wood carving still requires multiple tools and is akin with
something like e.g. distributed systems, not language/ product X.

------
ankurpatel
Why are you adding an extra level of abstraction in between Rails and Sidekiq?
Why not just have Rails push to redis queue and elixir process finishes the
job of sending email or doing what it has to do. Why have Sidekiq run at all?
It seems adding multiple layers as such will make it harder to understand,
debug and maintain in future.

EDIT: Also your example of polling on Redis queue using elixir is very
inefficient and makes your article not so credible in my eyes or eyes of other
good software architects. Redis is meant for pub sub and not polling.

~~~
wcummings
Redis has blocking list primitives which you can use to build an efficient
queue (push and pop in O(1)), if you look at the code you'll see he's using
brpop

~~~
ankurpatel
I agree but there is no need to poll. Polling is a poor mans solution in a
world where you can have persistent connection and get push notification of
when a new item is added to the queue and there is no need to throttle for 10
seconds before polling again.

~~~
danneu
A listening pattern still has to poll on boot for unprocessed jobs.

------
mmcclure
We've done similar things at work (using a Redis queue and a Go co-process),
but I actually really like the idea of using a fairly common and well-liked
delayed job interface in Rails and swapping out the worker underneath. Cool
idea.

One minor suggestion for the author (post doesn't have a comment section).
`:timer.sleep(10)` is explained as "sleep for 10 seconds", but sleep takes
millis as an argument
([http://www.erlang.org/doc/man/timer.html#sleep-1](http://www.erlang.org/doc/man/timer.html#sleep-1)).

------
Dirlewanger
Excuse my poor understanding of Ruby's GIL, but is the need for an Elixir
worker because Ruby is by default single-threaded, and therefore actual
default concurrent Sidekiq workers are not possible? Or this a problem scaled
up to the degree that we're talking ~thousands of jobs per second that Ruby's
limitations start to show?

~~~
phamilton
Since Ruby's GIL means you can't do true parallel processing, it's common to
kick off multiple worker processes and have them each take one job at a time.

If the job involves an expensive API call it will tie up a worker for the
duration of the call.

So it's not so much ~thousands of jobs per second as it is a limitation of
only 10 ruby processes per 1GB of memory. (A ruby process with the Rails env
loaded is about 100MB.)

Elixir gives you the ability to make a few orders of magnitude more calls in a
tiny amount of memory. (100,000 Elixir "workers" can be had in less than 1GB).

------
lshemesh
I recently did a Sidekiq background job in ruby passing messages to a python
script using zeromq. It was a blast.

------
spot
Polyglot web IDE that allows you to mix Python, Javascript, R, Ruby, Scala,
Java: [http://BeakerNotebook.com](http://BeakerNotebook.com) work with them
all in the same page. As a corollary, you can work with Python2 and Python3 in
the same notebook, and share data.

I totally agree that each language has its strength and each facet of your
problem may best be solved in a different language.

------
dozzie
Yeah, be polyglot by speaking American English and British English! I mean,
Ruby and Elixir!

[http://satwcomic.com/multi-language/moby-is-a-
dick](http://satwcomic.com/multi-language/moby-is-a-dick)

~~~
dagw
While Ruby and Elixir have quite similar syntax, their underlying run time
models are fundamentally very different.

~~~
pmontra
Yes very different model, and you realize that the syntax is not so similar
once you really start working with Elixir: you start using constructs that
don't belong to procedural languages at all.

Similarities don't go much further than def do end (but there are too many do
and too many different def in Elixir) and the sensible naming of some
Library.functions() to match classes and methods in Ruby's standard library.
But hey, making a good first impression is extremely important and Elixir has
been engineered well in that regard. It's not one of those I'll-never-ever-be-
able-to-read-it languages.

~~~
sanderjd
Yeah, I wonder if Elixir's superficial similarity to Ruby was actually a
mistake. It makes some people think they can just write Ruby and have it run
faster, only to become frustrated, and others scoff at it due to prejudice
against Ruby. On the other hand, I think it has a really nice syntax that
works very well for its semantics, so from a purely technical standpoint it
was a good choice.

~~~
phamilton
This is an open acknowledged issue on elixir-lang-core mailing lists. Many
requests to make Elixir behave like Ruby are shot down. Often, what is
requested already has a powerful analog in Elixir. If not, a lot of thought
goes into how to add said feature according to Elixir idioms and style rather
than blindly porting from Ruby.

