Hacker News new | past | comments | ask | show | jobs | submit login

Store all the assets/data however you like, then append a job description (referencing the assets/data) onto a queue. Separate servers have a worker process running that tries to fetch a job from the head of the queue and start processing, taking as long as it takes. When it's done, it fetches another. A separate overlord process monitors the queue health. If the queue grows too large, it fires up more servers. When the queue is small, it shuts down idle servers.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact