
Ask HN: How can I have a group of service peers assign themselves a number? - frankify
I have a set of a few hundred thousand xml feeds that I want to poll every XX minutes.<p>I&#x27;m writing a service in Go that pulls the list of feed URLs from a database and fetches some subset of those feeds.  This subset are the ones that this service will be responsible for fetching.<p>I&#x27;d like to divide up the work by simply numbering each server&#x2F;instance of this service I have running, and dividing the ID of the Feed by the index of the service modulus the number of peers.<p>What is a robust way that I can have these services discover their peers and reliably assign an index to themselves?  So, if I deploy 3 instances of this service, they will eventually discover each other and assign indexes of 0-2 to themselves.<p>I&#x27;d like it so if I add a new instance of this service, or if one goes down, the peer discovery mechanism will notice this and the services will re-assign a new index to themselves and then re-fetch their Feed list based on their new ID.<p>Is this a use case for something like etcd or Consul?<p>My initial thought was to have each service generate a random ID&#x2F;number and then each peer would register with something like etcd.  It would then monitor some key for peers and assign itself an index based on the sorting of its key relative to its peers.  I&#x27;m not sure if this is the best way to handle this situation though.
======
jacquesm
Hash the resource to be fetched, farm the work out to a node or work queue
based on the last n bits of the hash. That way you don't need any
centralization, any node can compute the hash and come up with the same id for
the node to do the work.

Good luck.

