

Ask HN: How to distribute jobs to multiple workers (servers) - pythoncall

Situation: I have lot of large file conversions from many customers which needs to be executed. Currently, I have a dashboard created which POSTs to a remote server. The endpoint on the remote server will shell execute a python script to launch file conversion. When it finishes, the python script will POST back to the server running dashboard.<p>I&#x27;ve been looking at solutions like https:&#x2F;&#x2F;github.com&#x2F;irmen&#x2F;Pyro4 but I can&#x27;t help but feel like I&#x27;m reinventing the wheel here. What I would need is a way to launch long running tasks on a cluster of remote worker servers that would be aware of load balancing.<p>What task management framework is suitable for my case?
======
byoung2
The most basic approach is to have the post go to the load balancer endpoint,
and have a cluster of identical servers run the python script. You could also
have the post add the job to a queue (e.g. Amazon SQS, RabbitMQ, etc). Then
your cluster of servers can pluck items off the queue as needed.

