AWS makes a lot of the concurrency issues easy (and scaling). Basically you can use their SQS ("Simple Queue Service"), add tasks to it, and when the individual drones check out a song from the queue, it's no longer available for a set amount of time.
If the drone finishes the process completely, it removes it from the queue permanently, but if the drone fails, dies, whatever, after that time-out it gets bumped back into Queue for the next worker drone.
We use FFMPeg for conversion.
note: tracktrack.it, if you're curious about watermarking.