Hacker News new | comments | show | ask | jobs | submit login

Store all the assets/data however you like, then append a job description (referencing the assets/data) onto a queue. Separate servers have a worker process running that tries to fetch a job from the head of the queue and start processing, taking as long as it takes. When it's done, it fetches another. A separate overlord process monitors the queue health. If the queue grows too large, it fires up more servers. When the queue is small, it shuts down idle servers.



Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: