

The bandwidth of a Boeing 747 and its impact on web browsing - jgrahamc
http://blog.cloudflare.com/the-bandwidth-of-a-boeing-747-and-its-impact

======
jerf
Hey, everybody bitching about how close a call it is for 1TB, just make it
10TB. There. Problem solved. 100Mbps internet link now no longer even remotely
close. Stop nitpicking, it's pointless. As the amount of data to move
increases, moving physical storage around wins increasingly more, nitpicking
won't change that.

~~~
lloeki
The whole comparison is flawed/irrelevant anyway. Their point is valid
irrespective of the airplane comparison in that having CDN servers closer
indeed mean faster downloads, but I could have a perfect 100Mbps connection to
the building across the street and try to download a 1TB file, it would still
be insanely faster to travel to the other side of the road and lug the hard
disk back.

~~~
dandelany
Of course it's relevant... Any time you're considering moving a large chunk of
data to a remote datacenter, it makes sense to do the math and determine
whether it's faster/more cost-effective to move the data there digitally or
physically. Why do you think AWS allows you to ship them your portable
storage? <http://aws.amazon.com/importexport/>

------
dredmorbius
This was actually utilized at some point in Usenet's history to transfer posts
between the US and Australia, as I recall. Googling isn't turning up citable
references, so I'll call this apocryphal until verified. I do find traffic
stats showing ~4GB/day posts as of 1996
(<http://en.wikipedia.org/wiki/Usenet#Technical_details>).

Australia for a long time was a worst-case major economy networking situation
with few, slow, and high-latency links (and still suffers from high costs due
to limited links and telco monopolies).

For an understanding of how limited international data links were as recently
as the early 1990s, one of the key public data links between the US and Europe
at the time was a 9660 baud link according to a dated, but fascinating,
compendium of networking at the time (lost somewhere in the Krell stacks, I'll
dig a reference on request).

Jon Bentley's classic _Programming Pearls_ discusses the problem of
transferring graphical data images between Mountain View and Santa Cruz,
California, in the 1970s. The high-bandwidth, cost-effective, sufficiently
reliable solution? Carrier pigeons. Really.

~~~
oz
Wasn't there an RFC entitled "IP datagram transport via avian carriers", or
something similar?

~~~
brazzy
Yup, RFC 1149: <http://www.ietf.org/rfc/rfc1149.txt> Extended by RFC 2549 to
add QoS: <http://tools.ietf.org/rfc/rfc2549.txt>

But the best thing is that this was actually implemented at least once:
<http://www.blug.linux.no/rfc1149/>

~~~
vr000m
and there is DTN using IP over avian carriers:
<http://dl.acm.org/citation.cfm?id=1614234>

------
kator
In my day we called this Sneaker net.. The bandwidth of a floppy vastly out
paced modems back then.. :)

<http://en.wikipedia.org/wiki/Sneakernet>

Never underestimate the bandwidth of a station wagon full of tapes hurtling
down the highway. —Tanenbaum, Andrew S. (1996). Computer Networks. New Jersey:
Prentice-Hall. p. 83. ISBN 0-13-349945-6.

~~~
antidoh
"Never underestimate the bandwidth of a station wagon full of tapes hurtling
down the highway."

How far we've come. Now we use 747s as the ironic competitor to digital
transfer.

~~~
wazoox
A Volvo full of LTO-5 tapes still has a very respectable bandwidth. LTOs come
in 20 tapes boxes, about 10 litres in volume; a Volvo break with backseat down
fits about 1500 litres, so you should be able to fit 150 boxes of 20 tapes =
150 x 20 x 1.5 = 4.5 Petabytes.

------
slyall
There are actually special file transfer protocols for high-bandwidth high-
latency situations where you want to transfer gigabytes/terabytes across the
world.

The main example I've heard is special effects companies like Weta Digital in
New Zealand wanting to deliver video to the US across a 200ms latency link.

There are a few open protocols in various stages of development or you can pay
for a product like FileCatalyst <http://www.filecatalyst.com/> ( some
interesting case studies on their website ).

~~~
efuquen
"There are a few open protocols in various stages of development"

Can you, or somebody else please name some? It always irks me a little when
someone just mentions that "these projects" exist, and then fail to mention
any of them.

~~~
EvanAnderson
We had a discussion about this over on Server Fault
(<http://serverfault.com/q/111813/7200>) and UFTP
(<http://www.tcnj.edu/~bush/uftp.html>), UDT (<http://udt.sourceforge.net/>),
and Tsunami-UDP (<http://tsunami-udp.sourceforge.net/>) all came up as viable
open source offerings (along with FileCatalyst and Aspera in the commercial
space).

------
Jabbles
This is why Amazon allows you to physically send them disks to upload data.
<http://aws.amazon.com/importexport/>

------
itsmeduncan
I came here to say that my father works for a company called MTU Areo Engines
as an FAA RS-DER. What that means is he can write and design repairs for the
hot-sections of most engines. Why this is interesting, and related to this
story is something that comes along with that.

All engine data that the pilot can see is available realtime on the ground at
the repair stations.

What this means is that an engineer can call up a GE-CF6 in the second
position on the starboard wing and see how it is performing. While that might
not sound all that interesting, there are literally tens of thousands of
measurements taken a bunch of times a second and transmitted to the ground for
each engine.

For instance, if the flight was 10 hours (and every package was 100 Kb) they'd
send about 35GB of data per engine. This is just the engine measurements. This
does not include avionics, communication, position.

Most of the information is propriety for how they store, transmit, and
calculate these things so I do not know exactly how this works. I have seen
one of these workstations and it is really impressive to know the exact RPM of
the number two engine compressor is spinning at in real-time of a flight
crossing the Atlantic.

Also, you can two-way text-message with the pilot from this workstation to
pass perceived information about the engine performance.

~~~
ramchip
FAA RS-DER: Federal Aviation Administration Repair Specification Designated
Engineer Representative

GE-CF6: A model of turbofan engine

------
shabble
I wonder how suitcase-full-of-truecrypt-disks compares to pgp/tls transport of
data over the internet in terms of probability of detection, interception and
access.

I recall the US now has a relatively free hand in confiscating storage media
for analysis/cloning during border crossings, and you might not get your copy
back for a couple of months if they take a serious interest.

So, even though the aeroplane travel infrastructure itself is quite reliable,
there are points of failure that would require another copy of the data to be
created and shipped, with a much greater latency (It might take days to
purchase more disks, transfer all the data onto them, find a willing courier,
book their flight, etc).

------
Toucan
A friend at a botanical institute sends and receives a 500GB hard drive
weekly, exchanging data across the Atlantic. Back of beermat calculations at
the time demonstrated huge efficiency compared to similar cost broadband
solutions!

------
epaga
Never underestimate the bandwidth of a station wagon full of tapes hurtling
down the highway. — Andrew Tanenbaum

~~~
masklinn
Eyup, not sure why this blindingly obvious truth has to be rediscovered once
in a while.

This is also how astronomers work, radiotelescopes produce terabytes of data
per day (the LCH produces ~15PB/year) so sending that over the wire makes
absolutely no sense. Instead, they ship external disks for smaller datasets[0]
and for bigger ones some go as far as shipping complete NFS/CIFS servers[1].

[0] [https://science.nrao.edu/facilities/evla/data-
archive/data-s...](https://science.nrao.edu/facilities/evla/data-archive/data-
shipment)

[1] <http://queue.acm.org/detail.cfm?id=864078> middle of the article

------
mbreese
This also makes the assumption that you'd have zero time copying the data once
you arrived in London. More realistically, you'd probably make a copy. In this
case, you need to also take into account the amount of time copying the files
at the other end. Depending on your external drive's interface, this is a non-
trivial amount of time.

~~~
reitzensteinm
Well if we're speaking more realistically, you'd use this for shipping 100tb
worth of disks (which would fit in your luggage in both size and weight), and
it would blow away just about any internet link you could lay your hands on
with a remotely similar budget.

That is a great point though. A 4TB drive, hooked up to your internal SATA
port, will do about 130 megabytes per second, which is around 9 hours for a
full copy.

Drive speeds increase linearly with track density, but drive capacity
increases quadratically, so this is only going to get worse. Hard drives are
the new tape.

~~~
whatusername
For a point of reference -- LTO5 pushes data at up to 140 megabytes per
second. (But Tape has the same problem -- Capacity increases faster than
speed)

------
antihero
> Boeing 747 has a bandwidth of 1 TBps.

Hmm, not sure about those numbers.

Say we limit it to this trip, at 10 hours.

A Boeing 747 can carry ~30,000 kg
(<http://www.boeing.com/commercial/747family/pf/pf_facts.html>) You can get
get 3TB 3.5' drives now, and they weight about 1kg
([http://www.amazon.co.uk/WD-Caviar-Green-WD30EZRX-
SATA-600/dp...](http://www.amazon.co.uk/WD-Caviar-Green-WD30EZRX-
SATA-600/dp/B0067LA6CS)), so 30,000 _3TB is 90,000TB in 10 hours.

10 hours is 36000 seconds, so that means that the speed of the data on a 10
hour flight is actually 90000/36000 = 2.5TB/sec.

That said, if we were to account for the next generation jump
([http://www.techspot.com/news/47860-seagate-60tb-hdds-
possibl...](http://www.techspot.com/news/47860-seagate-60tb-hdds-possible-
with-next-generation-storage-tech.html)) so 60TB for about a kilo, we'd get
60_30000 / 36000 = 50TB a second.

Obviously, the longer the flight, the more inefficient this transfer becomes.

~~~
jgrahamc
I'll modify the post to make that point. You are correct that the real
bandwidth of a single 747 is more like 220 Mbps.

~~~
brk
In either case, it's an interesting exercise but missing some key
calculations...

When I transfer data from machine to machine, the data is immediately "usable"
on the receiving machine.

When transferring data by physical means, the data needs to be brought to the
final point of consumption, and possibly loaded into the host machine if it is
not in some form that could be immediately connected/mounted.

So, the real comparison would be transferring data from the host machine to
some form of removable media, taking that media to the airplane (early enough
to meet all the pre-clearance stuff), waiting for the plane to takeoff, fly,
land and then handoff the physical media at the other end.

In many cases, I'm sure the plane is still faster, but if we're talking about
its theoretical bandwidth we need to look at the entire "trip" of the
packets/data, not just one segment of it.

~~~
excuse-me
Unless you take the rackspace crates off the 747 and hook them up at the
destination end. But compare that to ordering blank disks, having them
delivered, configuring them and then copying network data onto them.

------
jentulman
I'm assuming 747 option is using carry on luggage otherwise at one stage the
underlying infrastructure changes to packet transfer by baggage handler which,
in my experience, can be an incredibly high latency protocol with a
possibility at times of zero error correction.

------
vr000m
There is whole field of study called delay/disruption tolerant networking
(<https://tools.ietf.org/html/rfc4838>), which uses data mules or physical
mobility to overcome connectivity outages in constraint environment
([http://en.wikipedia.org/wiki/Routing_in_delay-
tolerant_netwo...](http://en.wikipedia.org/wiki/Routing_in_delay-
tolerant_networking)).

However, there is one thing that the delay (in the cited article) does not
take into account is that the flights do not take-off all the time, so if you
have data ready to be sent and no one to take it, then there is waiting delay
before it can be sent. so if there is only one flight in a day then some data
will have to wait between 24+10h to 0+10h before it can arrive. Since storage
cost is very low, one could ship a lot of data but this data cannot be delay
sensitive else.

on the other hand, using TCP one can prioritize data and send more important
data before the rest. Of course there may be intrinsic problems with TCP
receiver window and one of the proposal is to increase the initial window but
the research community and the IETF is a bit split on the real benefits of it.
The fear is that it may cause a congestion collapse. Moreover, the current
buffer-bloat in the routers and the increase in the window size may only aid
in causing a congestion collapse ([http://tools.ietf.org/html/draft-gettys-
iw10-considered-harm...](http://tools.ietf.org/html/draft-gettys-
iw10-considered-harmful-00)).

------
Tooluka
28 crashes a day is a lot. But author assumes that storage magically appear
inside 747 during takeoff add disappears during landing. But if we will
include this strange institution called customs then 28 item loss per day
won't be unrealistic at all.

Items detained indefinitely in the customs, items detained until bribe is
paid, items lost, items stolen during load/unload, items broken (hits,
pressure changes, magnetic gateways and handheld scanners etc.). Lots of
problems in general, all are multiplied if your country customs and airport
personnel are incompetent.

------
georgebarnett
While it makes for a good blog, this idea ignore cost ($1200? to transfer 1tb
data) and it ignore that you still need to get the 1Tb _off_ the drive once
you get to London.

And you need to sit in London traffic.

~~~
chayesfss
Typically drop shipping something on an airline is substantially cheaper than
a full fledged seat fare. You go to the counter, say you want to ship
something, they toss it in the belly and your friend picks it up at the
counter at the destination. We used to do this all the time with equip in
Alaska.

------
napolux
Old example based on the one in Tanenbaum's book (a little van against a 747,
but that's the same...)
[http://books.google.it/books/about/Computer_Networks.html?id...](http://books.google.it/books/about/Computer_Networks.html?id=Pd-z64SJRBAC&redir_esc=y)

------
spinchange
I enjoyed the premise behind this post, and perhaps this is pedantic or I am
simply confused, but why does the it conclude with, "The speed of light really
is a limiting factor in downloading!" and talk about what Cloufare does to
"fight the speed of light" after smartly explaining network latency at the
layers _above_ the physical one?

I guess fighting the speed of light sounds cooler, but that's not really the
problem they are solving or speaking too, is it?

*edit just for fun: 5MB data storage transfer, 1956 <http://mlkshk.com/p/AW8A> (img)

------
EvanAnderson
All this makes me think of is "In order to transmit the same amount of
information on paper, they would have to arrange for a 747 cargo freighter
packed with telephone books and encyclopedias to power-dive into their unit
every couple of minutes, forever."

~~~
alanfalcon
One of the wonderfully vivid images that so enamored me with Snow Crash as a
young college student.

------
angdis
OK... but the nice thing about computers and networks is that you don't
_necessarily_ have to transfer 1TB globs of data to from SF to London to use
it or compute upon it. There are computers in SF too. And of course, if you're
talking about media files you can't watch more than one at a time and anyway
you don't need "random access" to the media file (you can stream them).

------
wtvanhest
Does anyone know why the system doesn't send over 10 packets at once, then
check those packets, then ask for the packets that are not received.

Wouldn't doing it that way cut down on the latency signifigantly?

(or send the entire file, 1000s of packets) determine which don't make it and
just ask for those?

This isn't my area of expertise, so I'm just curious.

~~~
jgrahamc
Two things:

1\. TCP wasn't designed like that. It uses a rather simple scheme of sending
back a byte count telling the other side up to what byte number it has
received. So there's no provision for telling the other side that a bit in the
middle is missing.

2\. We don't send 1000s of packets at once because doing so would cause
congestion on the Internet. So there's a whole subsystem in TCP that tries to
determine the capacity of the link between two points and avoid creating
congestion.

3\. But the scheme you describe was actually added to TCP and it's called
SACK:
[http://en.wikipedia.org/wiki/ACK_(TCP)#Selective_acknowledgm...](http://en.wikipedia.org/wiki/ACK_\(TCP\)#Selective_acknowledgments)
Note that the original article isn't talking about the case where packets are
being lost.

~~~
ajross
Note that SACK is enabled by default on all modern TCP implementations, so #1
really isn't an issue.

And the 1000's of packets thing isn't quite right either. With windows scaling
(another option pervasively enabled) and a fat pipe (it does take some time
for the algorithm to ramp up that far), you can have about 1GiB in flight on
the wire at any time, that's hundreds of thousands of packets.

------
rdtsc
This calculation works if you can directly plug in the disk and start using
the data that way without copying. If, when the disk arrives, you still have
to plug it in and copy it at 50MB/s via an IDE cable then that has to be
factored into the calculations.

------
quarterto
_One thing British Airways ensures while flying the 1TB of data is
reliability._

Modulo baggage reclaim, of course.

------
kokey
The analogy would have worked better if you used a Fedex jet instead of a BA
one. That said, I don't think the latency of transferring large data is the
best selling point for Cloudflare, especially since the acks are asynchronous
and good load balancers perform quite well over long latency links.

A better selling point for distributed delivery is the round trip when you
deal with multiple transactions, for example requesting a web page and then
the included objects, or submitting data and waiting for a response.

------
rexreed
Given that the OP doesn't factor in delays due to transit to and from the
airport, clearing immigration and customs, and other delays, a more apt
comparison would be the relative bandwidth of transferring 1 TB from one part
of town to another. At that point, you're only facing traffic. If it takes 1
hour from one part of the a town to another part of the same town (4 if you
are in LA), then the relative bandwidth is much more remarkable.

------
ae7
Two things nobody is considering: price and time to transfer data to the
disks. I'd rather pay for a 100Mbps connection than 10 hours of jet fuel.

------
chayesfss
See, I always told my friends it was faster to burn to CD and then toss the
disc their way. I mean, the last time I had to make a null modem cable it took
me hours just to find the right tools!

~~~
GreenNight
With friends we do the same when we meet to LAN play. We copy the games into a
USB stick and pass it around. It's waaaaay quicker than downloading from steam
_, and quicker than copying through the local net.

_ If you know the right folder you just copy it there, buy the game if you
didn't already have it, and just wait a little for it to check the integrity
of the install.

This article has me almost wishing to start a data courier service. Flying
around the globe transporting data.

------
iamben
Reminds me of the broadband vs carrier pigeon story a few years back -
<http://www.bbc.co.uk/news/technology-11325452>

------
goblin89
What about environmental impact of delivering data via plane vs. cable? It's
probably too complex to compute precisely, though.

------
georgecmu
s/bandwidth/throughput/g

------
excuse-me
Easy to beat a 747

A rackspace portable storage center fits 12,000 hard drives (=36Pb) of data,
into a 40ft shipping container.

A container ship like the Emma Maersk can carry 15,000 TEU (20foot containers)
across the Atlantic in 5days

So 12,000drives * 3Tb * 7000 containers = 250 million Tb, and it takes 5days =
450,000 seconds

So a data rate of 250,000,000Tb/450,000s = 600 TBytes/s

Although you do have to watch out for pirates!

~~~
Retric
A Boeing 747 LCF <http://en.wikipedia.org/wiki/Boeing_747_LCF> can cross the
Atlantic in 3.5 hours and carry 180,530 kg.

3TB drives weigh ~1 lb = 180,530kg/1lb * 3Tbytes / 3.5 hours in seconds ~=
235TBytes/s which is rather close. Especially when you consider how long it
takes to load those container ships and how limited the destinations are.

PS: As to pirates your 84 million drives at 150$ a piece would be worth ~12.6
billion dollars which might actually be a tempting target even if you can only
get 10% of their actual value in ransom.

Edit: Even this is small next to MicroSD cards. ~64GB/gram = 29Tbytes/lb =
2,270Tbytes/second across the Atlantic for ~12.64 billion.

~~~
tcpekin
How did you do get your result? I calculated 758 Tbits/second [1] or 95
Tbytes/second [2]. I think you or I may be missing a conversion somewhere.

[1]
[http://www.wolframalpha.com/input/?i=180%2C530kg%2F1lb+*+3Tb...](http://www.wolframalpha.com/input/?i=180%2C530kg%2F1lb+*+3Tbytes+%2F+%283.5hours*3600sec%2Fhr%29)

[2]
[http://www.wolframalpha.com/input/?i=180%2C530kg%2F1lb+*+3Tb...](http://www.wolframalpha.com/input/?i=180%2C530kg%2F1lb+*+3Tbytes+%2F+%283.5hours*3600sec%2Fhr%29+in+terabytes%2Fsec)

~~~
Retric
Heh, I just did it again and got your numbers so I think I made the mistake.

PS: It's off by ~2.4 so probably did the kg -> lb conversion twice + some
rounding errors.

Let's see if I can redeem myself Heathrow Airport averages a little over 1300
flights per day. If 1 flight can handle 1.2 million TB's
[http://www.wolframalpha.com/input/?i=180%2C530kg%2F1lb+*+3Tb...](http://www.wolframalpha.com/input/?i=180%2C530kg%2F1lb+*+3Tbytes+)
then Heathrow Airport in theory is a router than can handle 17,965 terabytes /
sec
[http://www.wolframalpha.com/input/?i=180%2C530kg%2F1lb+*+3Tb...](http://www.wolframalpha.com/input/?i=180%2C530kg%2F1lb+*+3Tbytes+*+1300+%2F+24+%2F+60+%2F+60+in+Tbytes)
(Not that they are setup to handle freight, but at least flight time is not
important only throughput.)

~~~
excuse-me
But the rackspace containers you just need to hook up power and water - then
you are done!

------
dr42
This doesn't take into account UK customs. I recently send a nicely packaged
1Tb drive to my place in tb uk by express mail from San Francisco. The drive
arrived in customs two days later, sat there for a week, then they mailed a
postcard saying the import duty was 550 GBP. Now we are up to about the third
week... Obviously I didn't pay that so the drive went back, taking over two
months. Using scp would have been far quicker...

So next time you're considering physical transport include the time spent in
the destination country's customs, with their highly incompetent and
inefficient crews.

------
lucb1e
I find this a rather misguiding post.

First off, do you really think it's more practical to ship data per airplane
than a simple cable that's already connecting you? Not that bandwidth is
completely for free, but most of the time you're paying a flat fee and
transferring this 1TB won't cost you anything extra. I bet that Boeing is
quite a bit more expensive.

Secondly, at some point the article assumes the 'transfer rate' of the Boeing
to be 222Mbps. While this might be true for this practical example--let's
ignore that you actually need to get the data on the plane in the first place
--I bet that you could load it up from bottom to top and reach practically a
speed of many gigabits ifnot terabits per second.

Thirdly, the Boeing flies with physical stuff. If the plane crashes, the data
is lost. You would need backups, which take time. Over the internet it's not
even possible to transfer the original medium; you'd need to first transfer
and then delete it.

Fourthly, while it might be true that a packet gets corrupted 28 times during
this 1TB transfer (I myself do not have statistics on this, but let's assume),
on the internet you can retransmit a packet. Try retransmitting that one
sector on the harddrive in under a second.

Fifthly, the article goes on about how TCP needs to wait for an
acknowledgement (or 'ack' in short). I've made this mistake before when
choosing for UDP instead of TCP because "it would not have to wait for acks
and thus make my game much faster". It's wrong because there is a so-called
congestion window. I do not know the exact figures, but it's like this: The
server sends packet one. Packet one is in transit. The server sends packet
two. Packet one and two are in transit. The server sends packet three. Packet
one arrives. Packet two and three are in transit. The server sends packet
four; the client sends ack 1. Packet 2, 3, 4 and ack 1 are in transit. Etc.
The thing is that the server can send up to X packets ahead, without receiving
an ack.

Sixthly, the article finally concludes that this is why you should choose for
Cloudflare--so it was an advertisement after all. That explains why it's told
in this story-like way and why it contains so many technical errors.

~~~
jgrahamc
To address some of your points:

1\. Yes, physical shipping is more practical than using the Internet for large
transfers and this is done frequently. For example, Amazon Web Services offers
the possibility to send them disks: <http://aws.amazon.com/importexport/>
Shipping of disks is also common in the movie industry. For example, a lot of
digital movie distribution to cinemas relies on shipping of hard drives.

2\. I used a statistic of 0.1% packet loss on the Internet and 28,000 flights
per day to arrive at 28 flights lost per day.

3\. There are two interacting things happening in TCP: the congestion window
(which limits the number of outstanding _packets_) and the receive window
which limits the number of outstanding bytes. This blog post does not address
the congestion window at all and I assume that the receive window can always
be filled. Thus is _overestimates_ what might be possible on a real link.

4\. For very large transfers there are improvements that can be made using UDP
and there are protocols that do just that. See, for example, TixelTec.

~~~
masklinn
> Shipping of disks is also common in the movie industry.

As well as sciences based on massive amounts of data (astrophysics and
particle physics for instance, where experimental systems can produce
terabytes per day), financial data analytics, or petroleum seismic surveys.

