
Why Zapier Doesn't Use Gevent Yet - mikeknoop
https://zapier.com/engineering/why-zapier-doesnt-use-gevent-yet
======
obeattie
The outbound dispatcher here seems like a prime candidate to write in a
different language (Go for instance). Presumably it just makes HTTP requests
so can be very simple -- building the request payload for instance could still
happen in Python. And as you use Celery you already have a message queue over
which it could communicate with the rest of your application.

~~~
sametmax
There is not need to overdo it. Asyncio + threads will do nicely. 500 req/seq
is not the end of the world. I have similar load on purely threaded systems
and they work fine.

Crossbar.io would help though, but is not even necessary.

------
iamd3vil
This seems like a perfect case for something like Elixir/Erlang where you can
spawn multiple processes and need not worry because of immutability.

~~~
mateo411
I think that is what they are doing with celery. Each server has 92 celery
workers. Each worker runs in its own process. Each worker takes tasks off of a
queue and completes them.

------
miseg
As a complete tangeant, I find that Zapier's UI is what holds back my usage of
it.

They added folders which is fantastic.

But editing a Zap is slow in the UI. I can't open a Zap into a new tab by
Ctrl+Clicking the Zap title. Lots of Zap management is checking one zap's
setup, and adding that configuration in turn to another zap.

~~~
mikeknoop
That's a smart thing we should add. Any other feedback? If you'd prefer to
share privately: mike@zapier.com

~~~
sandGorgon
have you evaluated RQ vs Celery. There have been many, many posts (even here
on HN) about how managing Celery has been very painful.

what have been your experiences/protips ? And what's your opinion on something
like RQ

~~~
robgolding
We haven't evaluated RQ specifically, since RabbitMQ is a core part of our
infrastructure and we're really happy with it. We have experienced the pain of
managing Celery, and considered alternatives a few times though! A post about
our experiences there is on the list to write! :)

~~~
sandGorgon
Ahh... For being on Rabbitmq, I suppose there are few other options. Without
going into a blog post, why not Redis? Other than fairly sophisticated message
routing... Any other reason?

------
natmaster
"accessing a global variable"

Um. What? Why do you have global variables for your celery tasks? They should
be idempotent and small.

Also, even your http requests should be globally stateless as that is the
nature of http.

~~~
agopaul
I think he was referring to global variables inside the 3rd party libraries
rather than their codebase. If a library is not thread-safe they cannot use
it.

