You can maintain a population of arbitrary size when using approaches such as tournament selection. Each worker can pull a subset of the population from another datacenter and run an isolated tournament. You also don't have to run the entire population each iteration. You can keep win-loss statistics and handle final selection/crossover/mutation in a separate thread once enough have accumulated.
Put more simply, run it like chess and use things like Elo rating to determine fitness at selection time.
I don't think this is the case for diffusion models.