
Cloudflare Package Repository Mirrors - Foxboron
https://cloudflaremirrors.com/
======
james412
As much as I'd love to use CloudFlare's uniformly fast network for package
update, I'd never use this simply because I already depend on them for too
much stuff.

It's not like letting them gather logs on my computers updating would be
terrible somehow, but if it's so easy to avoid donating more logs and more
dependence on them, why not do it for the sake of preserving diversity?

Growing companies often exhibit a long term pattern that usually seems
innocent at first manifestation:

\- Offer something for free

\- Become hugely popular at offering that free thing

\- Alternatives slowly dwindle in popularity and/or begin to die off

\- User 'choice' increasingly becomes concentrated in that single option

\- Company introduces surprising policy change, but users can switch to an
alternative if they don't like it

\- Company introduces differentiating functionality setting its option apart
from the alternatives.

\- Users grow additional dependence on the new functionality, alternatives
look increasingly obsolete

\- Company becomes increasingly aggressive at introducing changes, as they now
own the primary alternative

It's hard to imagine CloudFlare are somehow going to monopolize package
management :) But nonetheless, free services like this generally signal the
beginning of this pattern

At some point I'm sure e.g. DejaNews seemed like an innocent dependency, and
that attempting to look far ahead into the future would not have revealed its
eventual sale to Google and the loss of the only comprehensive archive of the
early Internet community in existence.

~~~
bhouston
I think this is still a good service. It is just a mirror. It is a way of
giving back to these open source operating systems and cutting their costs.
Thus embrace it but be smart about it so that no one is trapped.

Dejanews isn't the same. That was an archive of a dying technology. This isn't
the same case. Usenet was a dying technology, ultimately killed by http.

Cloudflaree has always done the free thing to raise it's popularity. You are
right that as companies get bigger it can be harder to keep revenue growing
and they slowly start to cross lines they didn't before though. If that
company has built reliance on it's free tools which have reduced diversity,
when they start to cross lines it can be sort of a trap. This usually isn't
intentional now but it is just how things work... Years from now revenue
growth might require crossing a line...

------
SXX
Is anyone from CloudFlare or Canonical reading?

Please parter to make mirrors for Launchpad PPAs. They're so painfully slow
around the globe and there is tons of open source projects that depend on
PPAs...

~~~
Ayesh
I recently moved to Vietnam, and I sometimes get awfully slow connection
speeds to the mirrors nearby my city, to a point that switching to US mirrors
is faster. (ftp.[kr|sg].debian.org)

~~~
stingraycharles
How is Docker hub from Vietnam nowadays? I lived in Cambodia from 2014-2016,
and it was nearly impossible to pull Docker images which have large single
layers.

Would be great to have a Cloudflare solution for this.

~~~
momothereal
Off-topic. I visited Cambodia for 3 weeks in 2017. It's a beautiful country
and I enjoyed every minute. My mind has pondered over living there long-term
-- it seems so affordable.

How was your experience living there for a while, especially as a developer?

~~~
Ayesh
I surely miss the developer/tech atmosphere that was there in Berlin,
Frankfurt, etc.

The life is quite simple, and the coffee culture is based around chatting with
friends for hours, instead of sitting in a cafe and getting work done on a
MacBook Pro or a Thinkpad X1 carbon.

If you live like a local, which is not to say any lower living standards, it
can be pretty cheap. At around 2x the living cost of a local, one can live
comfortably here. I pay around third the rent which was once near Berlin, and
now spend about half on food if I eat out.

Internet is cheap but unreliable. I sometimes have to use my 4G mobile hotspot
(1GB/day at ~€4 a month).

I was born and raised in an Asian country, so there isn't much of a culture
difference to speak of. However, Vietnamese is pretty difficult to learn, and
even after six months here, I still cannot get the tones right and speak more
than 50 words. Indonesia in that regard is amazing.

If you are thinking to move to south East Asia, I can highly recommend
Indonesia. They have countless mountains to climb, a good hiking/riding groups
to easily join, similar living cost to Vietnam, and a much easier language to
learn. Bahasa Indonesia is understood pretty much everywhere, and quite easy
to learn. Internet is censored in both countries to some extent (Reddit
doesn't work in Indonesia with a DNS blackhole that you can easily bypass),
and has worse Internet in Indonesia in general though.

There are many startups based in Indonesia (such as Gojek, Traveloka,
Tiket.com), some of which are owned by SoftBank a few layers up. I feel it's
more easier to I best/startup in Indonesia too.

~~~
SXX
Where exactly in Indonesia have you lived? Bali is like two times more
expensive compared to Vietnam.

Also internet connection was generally much worse for me in Indonesia due to
their island nature I guess. In Vietnam it's super easy to setup two
50-100Mbps FTTB lines for something like $80 / month (just in case one of ISP
go down).

~~~
Ayesh
Yogyakarta, Malang, and Bandung (all on Java).

------
Pirate-of-SV
I ran rankmirrors (from pacman-contrib) on the mirror and the 5 upstream i got
on
[https://cloudflaremirrors.com/archlinux](https://cloudflaremirrors.com/archlinux)

rankmirrors test the time to download core.db from each mirror. Basically
this:

    
    
      curl -s -m 1 -w "%{time_total} %{http_code}" -o/dev/null https://cloudflaremirrors.com/archlinux/core/os/x86_64/core.db.tar.gz
    

The results vary a lot sometimes cloudflare mirrors is the fastest, sometimes
it's the slowest. Getting anything between 0.07 and 0.30

    
    
      $ rankmirrors -t ./ml
      Querying servers. This may take some time...
         *   *   *   *   *   *
       Servers sorted by time (seconds):
      https://ftp.lysator.liu.se/pub/archlinux/$repo/os/$arch : 0.113249
      https://ftp.acc.umu.se/mirror/archlinux/$repo/os/$arch : 0.120371
      https://mirror.osbeck.com/archlinux/$repo/os/$arch : 0.127601
      https://archlinux.dynamict.se/$repo/os/$arch : 0.180493
      https://ftp.myrveln.se/pub/linux/archlinux/$repo/os/$arch : 0.248406
      https://cloudflaremirrors.com/archlinux/$repo/os/$arch : 0.302824
      
      $ rankmirrors -t ./ml
      Querying servers. This may take some time...
         *   *   *   *   *   *
       Servers sorted by time (seconds):
      https://cloudflaremirrors.com/archlinux/$repo/os/$arch : 0.070069
      https://mirror.osbeck.com/archlinux/$repo/os/$arch : 0.081506
      https://ftp.lysator.liu.se/pub/archlinux/$repo/os/$arch : 0.098763
      https://archlinux.dynamict.se/$repo/os/$arch : 0.125846
      https://ftp.acc.umu.se/mirror/archlinux/$repo/os/$arch : 0.127948
      https://ftp.myrveln.se/pub/linux/archlinux/$repo/os/$arch : 0.136529

------
overflow_error
How is this supposed to work? (I use Arch BTW) supposed I'm in a country X,
does Cloudflare basically caching the request to server that are in country X?
But I'm already close to that server and I'm always cautious with mirrors that
are in country X, often they were out of date :(

~~~
gravitas
This is my issue with it as well - it's basically their geo-sniffing magic
routing you towards the closest officially listed mirrors for that country
(glorified proxy); it makes the mistake of presuming that all those mirrors
listed in the Arch country-mirrorlist are created equal in terms of
reliability and performance, but they're not (we have a whole web page just to
track out of date mirrors).

~~~
iforgotpassword
I'd guess if this gains traction, the caching on Cloudflare's side will
usually make up for this, and if it hits the actual mirror behind it, the load
on that one should be lower than before. It reads like they will try to proxy
multiple local mirrors so time will tell how well they will pick amongst them.
I'm switching for now to see how it will perform over a longer period.

~~~
gravitas
The difficulty is if the mirror itself is behind, Cloudflare's cache for that
specific endpoint will be correct (functional, working) but out of date
compared to another mirror.

Example: around the end of June this year the mirrors.kernel.org infra was
failing to sync (DNS issues of some sort) the latest content, so your upgrade
would appear to have no new packages because the stale content was a valid
repo, when in fact there were many updates and other mirrors were in sync.

~~~
iforgotpassword
Right that makes sense, so Cloudflare would actually have to make the caching
context-aware, not just classic HTTP proxying.

~~~
gravitas
I'm not aware of the technical how, but the Arch infrastructure seems to
measure this successfully; a hook into this data might be a great way for the
two to interoperate (so that Cloudflare always serves the freshest
cache/mirrors).
[https://www.archlinux.org/mirrors/status/](https://www.archlinux.org/mirrors/status/)

~~~
Foxboron
The relevant code can be found here I believe:
[https://github.com/archlinux/archweb/blob/master/mirrors/man...](https://github.com/archlinux/archweb/blob/master/mirrors/management/commands/mirrorcheck.py)

------
dennajort
Is it me, or it downloads so fast that the progress output doesn't even show
in the terminal ? On Archlinux.

~~~
rwha
I saw the same. My best initial guess is that this is somehow caused by
cloudflare either setting the content-type header to text/plain or not setting
the accept-ranges header. Or maybe both?

~~~
p00f
> not setting the accept-ranges header

Mostly that, because it doesn't support resuming partial downloads either (it
says http server does not support byte ranges)

------
dtf
I can't get it to work, packages upgrades show this error. Any ideas?

    
    
      error: failed retrieving file 'XXX' from cloudflaremirrors.com : HTTP server doesn't seem to support byte ranges. Cannot resume.

~~~
p00f
Clear your partial downloads using sudo pacman -Scc, another comment says that
it doesn't support byte ranges so it can neither display progress, nor resume
partial downloads

------
rwmj
It would be really nice if web proxies supported this as a generic mechanism,
which could be applied to Linux packages, ISOs, git commits, docker layers and
more. I wrote up a very half-baked proposal for this kind of thing a while
back: [https://rwmj.wordpress.com/2013/09/09/half-baked-idea-
conten...](https://rwmj.wordpress.com/2013/09/09/half-baked-idea-content-
addressable-web-proxy/#content)

Then you could set cloudflare as a proxy (or transparent proxy even?) and it
could inject the objects without having to visit the endpoints.

~~~
viraptor
Ideally soon repos will be mirrored to ipfs nodes and that will automatically
pull the hash-addressed content from the closest proxy, which may or may not
be cloudflare. The protocol is already there. Some initial solutions are
already there too: [https://github.com/victorb/arch-
mirror](https://github.com/victorb/arch-mirror)

------
rb2k_
Another set of mirrors with decent network infrastructure can be found at:
[http://mirror.facebook.net/](http://mirror.facebook.net/)

(Not holding my breath for the kind of responses this comment might get ;))

------
bhouston
Need Ubuntu and Raspian.

When I saw the title I though they were proxying npmjs or docker hub.

The install process of editing a file is a bit painful. Means few people will
do it. Somehow would be nice to have this offered in the is as a built in
optional feature.

~~~
emmelaich
Also Fedora and Centos. Please.

------
gravypod
I hope this is expanded to other package repositories (Maven, Docker, npm,
etc). I'd really like to make it easier for people at edge locations to build
my code.

------
cabraca
Now every time cloudflare has a hickup, not only is half the internet
unreachable, you can't even update your server... nice /s

~~~
YorickPeterse
At least on Arch Linux you can specify multiple servers, ensuring you don't
depend on a single one.

~~~
cabraca
I know, but lets be real for a moment: Cloudflaremirror will probably be the
fastest. This will cause people to configure it and because people are
generally lazy they will not configure other mirrors.

------
sneak
There are also other mirrors on cloudflare-ipfs.com.

If you are interested in other mirrors, perhaps you should IPFS pin some
yourself. :)

------
tiernano
Been looking for something like this for a while. We see if more repos get
added soon (Ubuntu and fedora would be nice!)

------
Thomaschaaf
Before opening the page I hoped that it would be an npm mirror which can be
used in case npm goes down.

------
p00f
I don't understand, is Cloudflare hosting their own mirrors or what?

~~~
Yeri
I understand they just proxy (and cache). They are still requesting packages
from the actual repository if they don’t have it cached.

But I agree — don’t really see the point. Why not just become an official
mirror?

~~~
Matthias247
Being a proxy and cache is their business. All it takes to set things up that
way is then a little bit of configuration that points to the actual origin.

I think if they would be an official mirror there would be a fair amount of
more work being involved. Eg you can’t just randomly wipe the cache anymore
and need to have higher persistence guarantees, since you are not supposed to
refetch from other mirrors all the time.

