
Dragonfly: Alibaba P2P file distribution system - eb0la
https://github.com/alibaba/Dragonfly
======
netvarun
Container image distribution seems to be one of the primary problems this
tackles:

"DevOps ... brings a lot of challenges: the efficiency of image distribution,
especially when you have a lot of applications and require image distribution
at the same time. Dragonfly works extremely well with both Docker and Pouch,
and actually we are compatible with any other container technologies without
any modifications of container engine."

FWIW, this was a similar problem that I tackled for Golang gopher gala
hackathon 2015 - a custom bittorrent based docker image registry POC.

[https://github.com/netvarun/docket](https://github.com/netvarun/docket)

Interestingly my problem statement was somewhat similar:

"Large scale deploys are going to choke your docker registry. Imagine
pulling/deploying a 800mb base image (for example the official perl image)
across 200 machines in one go. That's 800*200 = 160GB [EDIT: Correction thanks
to kingbirdy] of data that's going to be deployed and it'll definitely choke
your private docker registry (and will take a while pulling in from the public
image)."

~~~
oscargrouch
Devs need to be aware that bittorrent now(for sometime) have a DHT solution
that allows to have "mutable" slots with the crypto public address tied up to
it.

So if you own the private key you can write the payload, and just share your
public-key address to the people you want to share the payload with. In the
payload you can write a traditional immutable torrent manifest for instance,
which is in essence a public-key crypto based update system.

For a lot of cases i think it's a better approach than what IPFS and DAT
provides, because you dont care about it being a global address. All you want
is to share with a group of people, more in the p2p social/organic way.

I was playing with it once, using the libtorrent library and the main
bittorrent DHT, and it was a very nice experience.. it finds the payload
pretty fast when you think its a DHT, and you are working in pure p2p fashion.

The only single point of failure here is the DHT bootstrap peer.

Im planing to use this feature to distribute binary images for clients that
have my public key.

~~~
brokenmachine
Very interesting. Forgive my ignorance, but you mean it's kind of like mutable
torrents that only the uploader can modify?

Does a client just search the DHT for the public key? I thought torrent
clients searched for the hash of the files.

If it's searching for the public key then how does one person upload multiple
different torrents, or do they create a new public key for each torrent? How
does a client know which is the latest version if it has been updated multiple
times?

Are there any example projects using this?

~~~
rakoo
> Are there any example projects using this?

The only project I know is gittorrent
([https://blog.printf.net/articles/2015/05/29/announcing-
gitto...](https://blog.printf.net/articles/2015/05/29/announcing-gittorrent-a-
decentralized-github/)), but it hasn't gone anywhere.

------
ksec
Ok, this is huge.

>At Alibaba, the system transfers 2 billion times and distributes 3.4PB of
data every month, it has become one of the most important piece of
infrastructure at Alibaba. The reliability is up to 99.9999%.

I took at look at their repo and it turns out there are surprisingly lots of
good stuffs in it which never gets much spotlight or attentions.

~~~
oh_sigh
3.4PB/month is only 111,000 GB/day. That's not particularly huge in this day
and age, except to consumers.

~~~
Xorlev
I imagine this traffic is fairly bursty as well.

Also keep in mind this is a data distribution system. In the case of large
data pushes, new builds, etc. it's important that all peers get the new data
on a timely and reliable manner.

I suspect you're underestimating the problem this project solves by focusing
on a mostly irrelevant data rate stat.

------
pushrax
Quay has supported Docker image pulls over BitTorrent for a couple years now
[https://coreos.com/blog/torrent-pulls](https://coreos.com/blog/torrent-
pulls).

Docker themselves have discussed making the official registry extensible
enough to support BitTorrent pulls, but I don't know if anything ever happened
there.

Facebook has been using BitTorrent for deploys for something like 9 years now.
They configured the tracker to prefer sharing peers with longer matching
subnet prefixes, to keep bandwidth off the backbone as much as possible.

~~~
fizwhiz
Is there a paper or talk you could link to that describes their system?

~~~
jzelinskie
Justin linked the blog post[0] which is probably the best written description.
The short of it is that when you upload layers to Quay, it stream calculates
the BitTorrent pieces. Private layers are given unique swarms isolated by
namespace and peer discovery is protected by a tracker[1] that contains
middleware validating JWTs passed in the announce URL of the torrent. A custom
client[2] can be used to simplify downloading and importing of images into the
local docker CAS.

Honestly, most organizations don't have sophisticated enough networks that the
benefits outweigh the complexity of p2p orchestration. This is why it's
popular at Alibaba, Facebook, Twitter, but most people are still just using
the OCI distribution protocol[3].

Feel free to contact me (Keybase is in my profile) if interested. I'd love to
get more people on the path to p2p, but it's often a solution looking for a
problem.

[0]: [https://coreos.com/blog/torrent-pulls](https://coreos.com/blog/torrent-
pulls)

[1]: [http://chihaya.io](http://chihaya.io)

[2]: [https://github.com/coreos/quayctl](https://github.com/coreos/quayctl)

[3]: [https://github.com/opencontainers/distribution-
spec](https://github.com/opencontainers/distribution-spec)

------
rb2k_
From:
[https://github.com/alibaba/Dragonfly/blob/master/src/README....](https://github.com/alibaba/Dragonfly/blob/master/src/README.md)

\- supernode(Java)

\- dfdaemon(GoLang)

\- getter(Python)

Interesting distribution of languages in what seems to be a somewhat self-
contained project.

Oddly,
[https://github.com/alibaba/Dragonfly/tree/master/src](https://github.com/alibaba/Dragonfly/tree/master/src)
only contains the getter and the supernode at the moment

~~~
timdorr
Looks like dfdaemon is at the root:
[https://github.com/alibaba/Dragonfly/tree/master/dfdaemon](https://github.com/alibaba/Dragonfly/tree/master/dfdaemon)

------
betolink
This is very similar to what BitTorrent does or am I missing something?

------
baybal2
The amount of p2p-ness in it is not any much more than that of any DFS of
previous decade.

The P2P slang though is freaking everywhere here. There are P2P bank and P2P
brand sausages.

~~~
justinsaccount
Not quite, this is not a "distributed file system" this is a "file
distribution system".

This is more along the lines of
[https://github.com/lg/murder](https://github.com/lg/murder)

------
cabalamat
How does this compare to IPFS or BitTorrent?

~~~
noja
or Syncthing

------
spullara
Sounds like Twitter's Murder tool from 2010:

[https://blog.twitter.com/engineering/en_us/a/2010/murder-
fas...](https://blog.twitter.com/engineering/en_us/a/2010/murder-fast-
datacenter-code-deploys-using-bittorrent.html)

~~~
pm90
Looks like its not maintained anymore

------
akavel
Anyone knows if it does NAT hole punching? I'm interested in such a tool for
deploying to remote machines.

~~~
namibj
If you only need such a tool, other comments on this submission linked ways to
use bittorrent with docker, and µTP [0] seems to be reasonably good at
punching through the good old style of NAT, where ports are sequential and on
the same IP, with something that can coordinate accessible to both. It also
enables gentle use of your bandwidth, in the sense of playing reasonably well
even if you don't have fq-codel or similar in use on the router. With somewhat
nice networks it can be pretty gentle on other users of the networks, without
wasting any part of it. Do consider QoS though, it is preferable to send
other, important traffic first, as µTP is good at backing off in these cases.
The latency in backing off is just a little high to be stealthy towards
concurrent TCP connections. Packet loss is rare, but lag spikes are still a
nuisance.

[0]:
[https://en.wikipedia.org/wiki/Micro_Transport_Protocol](https://en.wikipedia.org/wiki/Micro_Transport_Protocol)

------
the_arun
Is Dragonfly a server or client? Cause it is compared with wget which is a
client. Am I missing something?

~~~
RIMR
It's P2P, so I would wager it is both.

~~~
synctext
It duplicates most features of libtorrent.org; but requires a server.
Therefore, a comparison to wget misses the 17 year old Bittorrent protocol.

~~~
nickpsecurity
Exactly. They should be comparing their P2P protocol to the whatever is
currently the best of open-source, P2P libraries.

------
SEJeff
Also a bit confusing as Dragonfly BSD is known for it's custom filesystem,
HAMMER:

[https://www.dragonflybsd.org/hammer/](https://www.dragonflybsd.org/hammer/)

~~~
RIMR
If you get confused between a filesystem called HAMMER, and a file
distribution system called Firefly, I don't know what to tell you.

They are different things with different names.

~~~
dpark
Well, this one is called Dragonfly, not Firefly, so the naming overlap is
relevant.

~~~
wccrawford
Do you also get confused by Dragon Naturally Speaking and Firefox and their
similarity to those 2 names?

~~~
dpark
These are literally two things with the name Dragonfly. I don’t particularly
care about whether the shared name is confusing here but it is disingenuous to
pretend that identical names are merely similar.

~~~
wccrawford
Dragonfly and Firefly are not the same word.

~~~
dpark
One of us is clearly missing something here. The two things I’m talking about
are Dragonfly BSD and Alibaba’s Dragonfly file distribution system. Neither of
these are named Firefly.

------
ofrzeta
Jesus, is it so hard to google at least once before naming a project? Just
recently that Sonar thing now this. What's next? A Windows editor called
Linux?

~~~
ofrzeta
To be fair, I've to admit that Dragonfly BSD doesn't show up on the first
result page when you search for "dragonfly". What does show up is the malware
by the same name.

Side note: this also shows how simplistic Google search really is. No way to
search for "Dragonfly /computers/" opposed to "Dragonfly /nature/" with the
terms in slashes denoting a concept or domain instead of a syntactic element.

~~~
agentd00nut
eh?

dragonfly +nature

dragonfly +computer

Though i don't blame you for not knowing, out of 4-5 operator cheat sheets and
guides i only see mention of the "-" to exclude terms.

~~~
ofrzeta
"+" is for synonyms. That might lead in the right direction but it's not the
same.

