

Ask HN: Job manager/queue for cloud instances - folli

I have following problem to solve and in order to spare me from reinventing the wheel, I hope for some pointers to existing solutions or services:<p>On top of a cloud infrastructure (AWS, Rackspace...) I need to somehow be able to manage jobs (research software which can take from a couple of minutes to a couple of hours to run), i.e. starting jobs on idle instances, spin up new instances if needed and stop unused instances.<p>Does anyone know of a solution or service which could help me in this regard?
======
jdale27
Please check out DNAnexus ([https://dnanexus.com/](https://dnanexus.com/)).
(Disclosure: I work there.)

It's a platform-as-a-service that allows you to do exactly what you're asking.
You submit jobs that you want to run, can set up dependencies between jobs in
a pipeline, and we do all the instance management (though you have the option
to choose different instance types to optimize the runtime or cost of your
jobs depending on their resource requirements). It's built on top of AWS,
though from the user perspective it's cloud-agnostic (e.g., you don't have to
know anything about how to bring up/down EC2 instances, nor do you even need
an AWS account).

Although it was designed with genomic data analysis in mind (in particular,
some of the relevant security controls for HIPAA and CLIA compliance), it's
actually quite general. So even if you aren't doing bioinformatics, you can
think of it as a nice service for running arbitrary Linux executables on
arbitrary files in the cloud with minimal setup and administrative overhead.

If you have any questions, feel free to shoot me an email (jdale at the domain
mentioned above).

~~~
folli
Thanks for the reply, I sent you an email.

------
bbgm
You might want to try CFNcluster
([https://github.com/awslabs/cfncluster](https://github.com/awslabs/cfncluster))
or Starcluster ([http://star.mit.edu/cluster/](http://star.mit.edu/cluster/)).
If you are used to Grid Engine, you should find them pretty easy to use.

~~~
folli
Starcluster seems to be what I was looking for! Thank you very much

