

How a Bug in My Ruby Code Cost Code School $13,000 - dyogenez
https://www.codeschool.com/blog/2015/06/04/how-a-bug-in-my-ruby-code-cost-code-school-13000/

======
shanemhansen
I'd add an additional safeguard. When retrying failed jobs, make sure to have
exponential backoff. Numerous queueing systems have this feature built in.

~~~
dyogenez
Good point! Delayed Job actually has an exponential backoff build in. It'll
run 25 times, with an increasingly large time between attempts before it gives
up.

The issue comes in when it doesn't know a job actually failed, as was the
issue in this case. Since the process was `kill -9`'d delayed job wasn't aware
the job failed and attempted to run it again as though it was the first
attempt.

