Hacker News new | past | comments | ask | show | jobs | submit login

Store all the assets/data however you like, then append a job description (referencing the assets/data) onto a queue. Separate servers have a worker process running that tries to fetch a job from the head of the queue and start processing, taking as long as it takes. When it's done, it fetches another. A separate overlord process monitors the queue health. If the queue grows too large, it fires up more servers. When the queue is small, it shuts down idle servers.

Applications are open for YC Summer 2019

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact