I love random numbers for distribution. I had a similar problem with a set of distributed clients that needed to download email, but only one client downloading at a time. The email servers also had an issue where a large number of emails in the inbox would cause the server to slow down exponentially. (eg. it didn't matter how many MB of email were in the inbox but it did matter if there were more than about 1000 emails)
The downloaders would download the list of inboxes to be fetched, randomize them and then lock the inbox when they started downloading, then the downloader would randomly pick a size cutoff for the max email size it would download, 10K, 1 MB, unlimited with a n inversely proportional maximum email count so that about 100MB could be downloaded at anytime.
We even had an issue with one server behind an old cisco router that barf'd on window scaling, so a few machines in the pool had window scaling disabled and that account would naturally migrate to those servers with window scaling disabled.
It worked wonders for distributing the load and keeping the Inbox counts to a minimum.
The downloaders would download the list of inboxes to be fetched, randomize them and then lock the inbox when they started downloading, then the downloader would randomly pick a size cutoff for the max email size it would download, 10K, 1 MB, unlimited with a n inversely proportional maximum email count so that about 100MB could be downloaded at anytime.
We even had an issue with one server behind an old cisco router that barf'd on window scaling, so a few machines in the pool had window scaling disabled and that account would naturally migrate to those servers with window scaling disabled.
It worked wonders for distributing the load and keeping the Inbox counts to a minimum.