
New improvements to IPFS Bitswap for faster container image distribution - yankcrime
https://blog.ipfs.io/2020-02-14-improved-bitswap-for-container-distribution/
======
jude-
Would be curious to know how using IPFS for internal container distribution
compares to using BitTorrent. IIRC BitTorrent has found similar uses in the
past.

Also, how well does BitSwap work when the underlying network is congested? Do
IPFS nodes do any kind of congestion control?

~~~
hiccuphippo
You can't update a torrent, if the content changes you have to create a new
one. IPNS helps with that. And you can't share pieces across different
torrents, if some still have the old torrent, they share it separately from
the new one even if the differences are minimal.

~~~
jude-
The article doesn't mention IPNS at all, nor does it talk about the need for
mutating an image while it is being shared, so I'm not sure why you think IPNS
is even desirable in this use-case?

~~~
toomuchtodo
"Inter-Planetary Name System (IPNS) is a system for creating and updating
mutable links to IPFS content. Since objects in IPFS are content-addressed,
their address changes every time their content does. That's useful for a
variety of things, but it makes it hard to get the latest version of
something."

[https://docs.ipfs.io/guides/concepts/ipns/](https://docs.ipfs.io/guides/concepts/ipns/)

TLDR IPNS are pointers to IPFS content (ie "latest"). If you're tracking your
containers and pinning to their versions elsewhere, might not need IPNS.

~~~
jude-
Thanks, but I already know what IPNS is. My points were that (a) it's not
needed for this use-case, and (b) it's a distinct system from IPFS. I think
you agreed in your TLDR.

------
pjc50
Netflix use IPFS? That's quite an endorsement and makes me take it a lot more
seriously.

~~~
Grustaf
”Netflix and IPFS began collaborating on ways to incorporate peer-to-peer
services into Netflix’s developer tooling”

Developer tooling is a pretty limited part of their traffic probably

~~~
bastawhiz
The volume of traffic doesn't really matter that much. Developer productivity
is just as important as the service you're actually selling when you're the
size of Netflix. If your N thousand engineers are suddenly unable to work, or
slowed down by X%, that's a huge problem. Large companies treat (or should
treat) developer tooling issues as seriously as application outages.

If Netflix is using IPFS for anything worth mentioning, it's almost certainly
substantive enough to be considered an endorsement.

~~~
Grustaf
Of course the service itself is more important. If there are outages in the
service, you will lose customers. If developers lose time at most your new
features risk delays. If developers are less productive over time you lose a
bit of money.

I say this as a developer for a FANG company.

It’s still an endorsement, but not nearly as strong as if the broadcasting was
somehow relying on IPFS. As it is, this is probably just some engineering
manager that made some non-crucial tool and put that on ipfs.

------
etaioinshrdlu
One big optimization that could help in some cases for container platforms
like Fargate is not downloading the entire image just to run the container.
Instead read files (or even just blocks) from network storage on demand.

This is basically how booting from a disk image works on most cloud platforms
too.

~~~
rochaporto_
That should be there soon :
[https://github.com/containerd/containerd/issues/3731](https://github.com/containerd/containerd/issues/3731)

------
js4ever
I'm shocked to see IPFS used to make something Faster...

~~~
DaniFong
it's remarkable! a bootstrap technique to improve time coherence -- what an
innovation!

~~~
viraptor
I think that was a dig at ipfs's issues with real world usage where a lot of
traffic until recently was wasted on metadata and every node used lots of
bandwidth to gossip. Meanwhile the actual throughout on a non-tuned node was
not great at all.

~~~
StavrosK
Has this changed now?

~~~
viraptor
I just realised that "recently" happened at the end of 2018
[https://blog.ipfs.io/53-go-ipfs-0-4-18/](https://blog.ipfs.io/53-go-
ipfs-0-4-18/) \- oops.

There was also a change later which turned off nodes being the middleman by
default, but not sure which version.

It's supposed to be much better these days, but I haven't tried it again in a
few months.

~~~
StavrosK
Ah, that agrees with my experience, I noticed the IPFS node behaving much
better some time in the last year.

------
miguelmota
It's awesome to see these kind of improvements on IPFS. A while back as a side
project I created an IPFS-backed docker registry which allows you to push and
pull docker images from IPFS [0]

[0] [https://github.com/miguelmota/ipdr](https://github.com/miguelmota/ipdr)

------
anonsivalley652
Yay. I last used ipfs for leeching abandonware around November. Although it
had a tough time getting started and it would occasionally freeze up for
several minutes, it worked well when it worked. It's seems to be getting
better from when I first tried it.

~~~
StavrosK
I wasn't aware of this use case, can you post a link/CID (if it's legal)?

------
hinkley
> The node sends out a want for each CID to several peers in the session in
> parallel, because not all peers will have all blocks. If the node starts
> receiving a lot of duplicate blocks, it sends a want for each CID to fewer
> peers. If the node gets timeouts waiting for blocks, it sends a want for
> each CID to more peers.

Trying to recall how the protocol works. Doesn’t this pattern of behavior mean
that a lot of machines will end up with the beginning of a file and few will
have the end? It sounds like the start of downloading would be very fast and
the end would slow down while it hunts for a source

May be why this is only 20% faster than Dockerhub.

~~~
tylersmith
Files are not downloaded sequentially, they're chunked into blocks which are
sent in parallel.

~~~
hinkley
Each client downloads in random order, or all clients download in the same
order?

------
tyingq
_" Web 2.0"_

Hadn't seen that gem for a while.

~~~
jungong
3.0, no?

~~~
411111111111111
pretty sure we're already at 4.0 with the Internet of Shit (IoT) and Mobile
Internet, depending on the one using the buzzword

but thats besides the point of the parent. web 2.0 hasn't really been
mentioned in ages.

------
tln
"the container runtime can be modified to retrieve layers identified by their
CIDs"

How do you do this? Exercise for the reader? :)

For the case of distributing containers in a datacenter with P2P, theres also
this work:

[https://github.com/uber/kraken](https://github.com/uber/kraken)

~~~
humblebee
This might be related:
[https://github.com/docker/distribution/pull/2906](https://github.com/docker/distribution/pull/2906)

------
psKama
I really don't understand why there is a big hype towards IPFS which is still
in development stage although, there are other options which are already out
and working like Sia - Skynet which is not even getting a fraction of
attention IPFS is getting.

~~~
Taek
Skynet is barely two weeks old fwiw. As more people play around with it and
see how strong it is I think it'll get a lot more attention. Lot of crypto
projects are already planning to add support in the coming months.

------
marshmellowtest
Saving Netflix's bandwith costs by sacrificing your privacy.

IPFS and bittorrent don't do anything to protect the data you are uploading
and your IP address.

Case in point:
[https://iknowwhatyoudownload.com/en/peer/](https://iknowwhatyoudownload.com/en/peer/)

Now every website you visit, any ad/tracker, any homecalling phone app can
tell what movies and contents you watch and when you are at home. For years.

~~~
nine_k
I thought that ipfs is about high availability, fault tolerance, including
some resistance against addressed censorship.

It never looked like an anonymizing tool to me; did anybody advertise it as
such?

~~~
marshmellowtest
"resistance against addressed censorship" does not work at all when all your
traffic is made public.

People can be prosecuted or otherwise harassed for sharing contents on a P2P
system.

> It never looked like an anonymizing tool to me; did anybody advertise it as
> such?

You are confusing "anonymizing" with "leaking a lot of information to the
whole world".

They constantly "forget" to tell people about the huge security impact.

~~~
viraptor
Ipfs helps you distribute content which may get taken down. It does not help
you evade local police.

For the second scenario, you want another layer which maintains secrecy. (Like
the tor transport
[https://ipfs.io/ipfs/QmYKQvBsbYrRhdaGvQXcEoSam7s5gKVYULfRgNP...](https://ipfs.io/ipfs/QmYKQvBsbYrRhdaGvQXcEoSam7s5gKVYULfRgNPzN5JM8N/IPFS-
via-OnionCat.html))

