

Murder: Fast datacenter code deploys using BitTorrent - abraham
http://engineering.twitter.com/2010/07/murder-fast-datacenter-code-deploys.html

======
naner
Sometimes I wish geeks wouldn't worry about giving something a cool sounding
name and instead just picked something conventional or unremarkable.

I don't want to search Google for help with Murder. I don't want to discuss
Murder at the office.

~~~
maukdaddy
or the next morning....did you complete the murder job?

~~~
samratjp
or commit murder (on github obviously)

~~~
Groxx
or find out who should be blamed for the recent Murder failure.

------
alec
I wonder if they tried multicast. It would be strictly more efficient in terms
of bandwidth (one copy per router/hop rather than all the different pieces
flying back and forth) and likely less CPU. There's premade software like uftp
(<http://www.tcnj.edu/~bush/uftp.html>) to do this.

~~~
moe
Amen. The NIH syndrome at twitter seems to be going strong. They just don't
stop reinventing stuff, and usually poorly.

In this case uftp is the time-tested solution, made _precisely_ for this
purpose (mass deployment) and provably better on all metrics (speed,
bandwidth, security via SSL, complexity).

The mind remains boggled...

~~~
nullstyle
You do realize that twitter didn't invent bittorrent, right?

;)

~~~
alec
They appear to have invested substantial time into integrating it with their
deployment system and improving its performance for their particular network
situation rather than using off-the-shelf software like uftp.

------
jcapote
You guys know a group of crows is called a Murder, right?

I'd hate to work somewhere where I could only use stuffy, politically correct
software names. Call me crazy, but quirky software names are half the reason I
release a project.

~~~
jolan
Indeed. I called my asset manager 'assman'. :)

~~~
mrduncan
Context: <http://www.youtube.com/watch?v=tujqM2u-BVo>

------
bonzoesc
Couldn't they use a tree-like distribution system where each node only
receives pulls from a small number of child nodes?

You could organize this by giving each server an incrementing number and using
math to deduce their parent node, much like an implicit binary tree stored as
an array: <http://en.wikipedia.org/wiki/Binary_tree#Ahnentafel_list>

That way you can still use the http or git protocol without having to invent a
special flavor of BitTorrent.

~~~
bjg
He covered why they didn't want to use a tree like distribution method in his
presentation.

It introduces a possible single point of failure for all nodes which are
children of a failed node.

------
DannoHung
Maybe this is too easy, but if you're using Git, why not have servers that get
a new release be added to a list of servers that can distribute a new release?

~~~
derwiki
Added complexity to continue to scale, and one deploy replica high up in the
chain that breaks will break everything downstream. As you start building a
system to address those issues, you're adding more complexity. This would lead
me toward a Murder based solution from the start.

------
maukdaddy

      no NAT/Firewall issues, no ISP traffic shaping, only trusted peers
    

Sounds like an easy-to-exploit datacenter to me.

~~~
winthrowe
They'd have those protecting the perimenter, but this is only for deploys in a
single datacenter right now. They're only doing it on their network, so they
don't have to worry about all the hoops torrents normally go through; they can
drop the timeouts to nothing, they don't need encryption, etc.

The video states that they will be looking into how best to do deploys across
centers in the future, but they are not going to have peers connect across
their wan links. One tracker and seeder per location is a first musing.

[Edit: paragraphs]

~~~
koenigdavidmj
>The video states that they will be looking into how best to do deploys across
centers in the future, but they are not going to have peers connect across
their wan links. One tracker and seeder per location is a first musing.

Heck, use a VPN to accomplish this if you're lazy.

------
metamemetics
css text-shadow on body content is over-rated and actually makes it harder to
read.

