

Ask HN: How to update a large network of computers - huherto

I am working for a company that has several thousand stores located in a large geographic area. Every time data changes have to be delivered to the stores (around 500MB each) there are about a dozen servers transferring the info to each individual store. As you probably imagine it takes a long time to do it and it requires a lot of bandwidth. It will be faster if the stores could be updated from other stores, specially if they are close from each other. Do you know of any software that can help us do it. May be some type of mirroring or maybe git or some other version control system work for it. Any suggestions?
======
bayareaguy
A friend of mine told me this was a Google interview question. Anyways, I'd
recommend taking a look at this:

 _The Frisbee server uses a custom multicast protocol to distribute highly-
compressed disk images to consenting clients. The Frisbee client is a multi-
threaded marvel that receives, decompresses, and smacks data down on your hard
drive, really, really fast. Frisbee can be used to install complete disk
images or single partition images._

\- <http://www.cs.utah.edu/flux/papers/frisbee-usenix03>

\- <http://www.emulab.net/software.php3>

------
noodle
<http://en.wikipedia.org/wiki/BitTorrent> (i would link to the protocol but
the link doesn't show up correctly here)

there are probably better options, but this one is already implemented and
likely wouldn't be that hard to make work for your situation. might require
you writing something yourself to suit your needs, or you might be able to
find a turnkey solution.

~~~
huherto
Ah, thanks I will look into it

------
st3fan
Some other thoughts:

You mention data changes. So what about sending just the changes?

What is the real problem? That it takes a lot of time? Or that it requires
bandwidth?

If downloads are slow, are your central servers overloaded? What about putting
the data on a CDN or Amazon S3?

~~~
huherto
The time.

~~~
st3fan
Sounds like you simply don't have enough bandwidth then. Fortunately that is
an easily solvable problem in 2008 :-)

------
st3fan
Assuming these stores are behind DSL ... DSL upload speeds are usually
terribly slow. Are you sure that it would actually be faster than downloading
from some central server?

~~~
huherto
I think so, even if the DSL upload is slower, we would not have the bottle
neck in the central servers.

------
bayareaguy
Here's another useful tool I just remembered.

<http://saf.bio.caltech.edu/nettee.html>

