
455 Tb of live streaming were transferred during the Olympic Games Rio 2016 - dreampeppers99
https://leandromoreira.com.br/2016/08/23/455-tb-of-live-streaming-were-transferred-during-the-olympic-games-rio-2016/
======
liquidise
> Globo.com only had rights for streaming the content to Brazil... with peak
> of 433K simultaneous users

This is a neat writeup but the bigger point here is highlighted by the opening
statements. While i get the massive revenue channel the olympics provides, i
can't help but feel that the media stranglehold over the events has gone too
far.

The olympics represent one of the only instances of global unity through
competition left in the world. Much the same way the world cup brings small
and proud nations into the global stage, the olympics has the same power.
Unfortunately, it is easier for me to find 20 year old simpsons clips on
youtube than it is for me to find a video of Bolt's 100m heat from last week.
Who benefits from this apart from NBC? I can list approx. 7.4B people who
don't.

As a lover of capitalism this all leaves me conflicted. But i would point to
the olympic coverage as the instance of corporations going too far with their
constricting of media sharing.

~~~
bdhess
I dunno, I found video of Usain Bolt's 2016 100m in less than a minute of
looking around on nbcolympics.com. Though I'd have to imagine that that
content is geofiltered, since AFAIK NBC only has streaming rights within the
US.

NBC paid the IOC something absurd like $12B for exclusive US media rights for
the games through 2032. Territorial broadcast rights agreements are a huge
source of funding for the IOC. It's hard to imagine how they'd maintain these
revenue streams if they abandoned the exclusivity aspect of their media
agreements, which likely implies that the games would suffer in some way.

~~~
Retric
I completely skipped watching the Olympics in large part because of NBC which
is terrible. So, I have some doubt that 12B is optimal for the IOC and it
clearly sucks for the rest of US.

~~~
aschampion
Watching via the BBC is a completely different experience: complete on-demand
archives almost every event, time bookmarks for groups/bouts/matches, no
annoying commentators unless the event was also being broadcast on a channel,
no missing most of the event because we need to watch a manufactured interest
piece on the US athlete, and you can go 30 seconds without hearing the name
Phelps.

Watching NBC's broadcast, in contrast, made it seem like NBC had no belief
that any of the events were interesting in and of themselves.

~~~
maxxxxx
I hate those sappy interest pieces! Just show me the event.

You are right. They seem to think that the event is not interesting.

~~~
mikekchar
I have historically avoided watching the Olympics because I like to watch
sports, not drama ;-). Having said that, NHK in Japan did a pretty good job, I
have to say. I watched the entirety of the women's rugby 7's semis and finals.
Great sports that I wouldn't ordinarily get to watch, and no Japanese
participation at all.

------
jordanbaucke
I'm interested in learning what the average "delay" for live-streaming would
be? At a high-level RTMP > (Segmenter: EvoStream) HLS > Cassandra < NginX-LUA
< HTTP Request (seems to flow model outlined here (From the 2014 World Cup):
[https://www.nginx.com/blog/globo-coms-live-video-platform-
fi...](https://www.nginx.com/blog/globo-coms-live-video-platform-fifa-world-
cup-14-part-ii-microservices/))

I'd assume it's minimum > 20 seconds depending on setup and teardown time for
first "chunk" in a sequence to reach cache and be transferred to a user?

~~~
zbobet2012
TL;DR: The delay is normally a segment length. Somewhere between 2-16s
depending on there configs.

Because the system is realtime (live) it _must_ not fall behind live. These
systems therefore are always only a single segment behind live. (A segment is
an independently decodable chunk of video, usually between 2 and 16s).

Source: I designed the backends, including segmenting, for NBC.

~~~
jordanbaucke
@zbobet2012 makes sense, certainly, you can shrink the chunk-size down to 2
seconds, I wonder if it's not practical to do though wouldn't the practicality
of setting up a setup > teardown of a new HTTP connection, fetching the next
chunk, etc. cause inconsistency in completing this procedure by the time a 2
second chunk has played out.

I believe people have used WebSockets to push these segments? Since once you
have established one TCP socket connection you don't have to setup a new
"session" for each discrete segment?

~~~
r1ch
HTTP keepalives prevent the excessive connection requests. The main issue with
HLS distribution is TCP sucks over high latency links, so you have to have
edges near your users.

~~~
jordanbaucke
Gotcha! I've also heard of people using WebRTC to reduce latency by setting up
a direct-to-user link between the ingest server and the end-user? Any idea how
this helps?

~~~
r1ch
Beam (recently acquired by Microsoft) does very similar to this. Rather than
wait for the whole X second segment to be built and distributed, they
apparently stream MPEG atoms through websockets which are them reassembled in
JS and presented via media source extensions. At this point you may as well
have re-invented RTMP, minus the Flash dependency :).

~~~
zbobet2012
Yep^^

In addition there is a major encoding downside to this. If your encoder can
grab a whole 2-10s chunk of video it can produce a better quality stream. This
is actually _super_ important for producing good video quality (especially on
things like sports streams).

------
tmcpro
Would be about $93k worth of bandwidth from AWS Sao Paulo Region (about 3x
more than AWS US-East)

~~~
aws_person
My deepest sympathies for someone who needs bulk, reliable networking out of
sa-east-1.

------
paulasmuth
None of these numbers add up.

If 30 million hours were watched with an average bandwidth of 2Mbps, the total
data volume should be 25,749TB (or close to 26PB) and not 400TB.

Also, if the peak bandwidth was 600Gpbs, that peak could not have lasted very
long. With that much bandwidth the claimed 400TB of total transferred data
would be used up in roughly 90 minutes.

So either the total data transferred or the number of hours watched figure is
off by a factor of at least 60x.

[ I figured "400Tb" was supposed to mean 400 terrabytes. If it's actually 400
terrabits, the numbers are off by one magnitude more ]

------
ryanlol
Wow, that's not very much at all.

In comparison, I've been using a bit over 600TB with a budget of less than
$500/month. With double the budget I could easily transfer olympic amounts of
data!

Really puts to perspective how incredibly accessible bandwidth pricing is
these days.

~~~
truth_sentinell
Do you have a _insert a dumb celebrity_ fan page? Or what do you do to consume
that bandwidth?

~~~
ryanlol
Scanning.

See for example:
[https://github.com/robertdavidgraham/masscan](https://github.com/robertdavidgraham/masscan)

------
ejcx
Curious how many Tb someone like twitch does per day. AFAIK they are the
biggest live streaming platform, since Netflix is all HTTP.

Seems like that could be a big moneymaker for them, for these big time events
that need a "live" aspect.

~~~
johansch
> since Netflix is all HTTP

You're saying you can't stream over HTTP?

~~~
niftich
Not OP, but the distinction is between live streams where the content is being
generated very close to the consumption time (historically delivered using
RTSP) and between static files sitting on a server/CDN (ie. Netflix, Youtube,
etc.)

~~~
ejcx
This is exactly the distinction I'm talking about.

Doing it live is way harder than moving static content out to all the leaf
nodes once per day and serving it to a geographically small region.

Adaptive bitrate has nothing to do with the point I'm making that, AFAIK,
twitch has the most "live" eyeballs out of anyone and there are special
engineering challenges there.

~~~
zbobet2012
Well, no Twitch isn't the largest. NBC/Comcast moved several orders of
magnitude more of live video for the last olympics, and those numbers went up
substantially this year+.

Live presents interesting challenges yes, but in some ways it is easier than
VOD from a scale perspective. In live all users are watching essentially the
same content++, so it is nearly perfectly cacheable in memory. That means you
don't really need a ton of cache, just a cdn with lots NIC's, ram, and CPUs.
You can use mid-tier caches to "fan" out data to edge caches and progressively
deload origins. You can take a look at our architecture over here
[https://www.bizety.com/2015/07/15/deep-dive-comcast-cdn-
arch...](https://www.bizety.com/2015/07/15/deep-dive-comcast-cdn-
architecture/).

The really hard problems with live are actually the _surrounding_ items. If
you are placing personalized ads, and 100 million people are watching a
stream, your system must support 100million transactions per second. If you
have QOS data from clients, you need to support 100 million TPS. Etc, etc.

Building live packagers and segmenters is also a serious bit of work.
Especially to "broadcast" standards. This requires the coordination of
multiple video streams subject to byzantine failure.

I guess the hardest scale problem though is probably emergency alerts. In this
situation all users will tune at almost precisely the same time to a live
streaming event. 30million + users can arrive on a single (previously cold)
item in < 100ms. Just opening that many tcp sockets is serious work even for
large numbers of servers.

\+ (sauces so I don't get in trouble, full disclosure I designed and work on
the systems that do this):

[http://nbcsportsgrouppressbox.com/2012/08/14/ondon-
olympics-...](http://nbcsportsgrouppressbox.com/2012/08/14/ondon-olympics-on-
nbc-is-most-watched-television-event-in-u-s-history/)
[http://www.fiercecable.com/special-report/judges-tally-
nbc-s...](http://www.fiercecable.com/special-report/judges-tally-nbc-scores-
historic-online-video-numbers-at-sochi-olympics-but-will-it)

++ [http://www.cs.umd.edu/~slee/pubs/iptv-
sigmet09.pdf](http://www.cs.umd.edu/~slee/pubs/iptv-sigmet09.pdf)

~~~
niftich
I'm actually not sure what's more intriguing or impressive -- the fact that
the [streamer for high-profile event] spins up so much infrastructure and
machinery for a duration of 2-3 weeks and then spins it down until the next
high-profile event two years later, or someone like Twitch who does it on a
smaller scale, but 24/7.

Though in my mind, the illusion or appearance of 'pop-up streaming' takes the
award. I wonder, how much streaming infrastructure and instrumentation and
code can someone like NBC reuse between two successive olympics (two years
apart)?

------
ilaksh
If its 455 TB of source video then you need around 91 5TB hard drives to store
that. Although they probably have a cheaper tape storage or something.

If you wanted to buy that many on Amazon you can get external 5 tb now for
$109. So that would cost about $9900. Which to me is a pretty small number
considering.

~~~
Animats
No, it's 455TB streamed out to all watchers, total, at an average bitrate of
2.0 Mbps. Far less unique content.

------
mmagin
Too bad multicast streaming never really happened.

~~~
zbobet2012
How do you think digital cable television works today? It happens, just on
closed private networks.

~~~
superuser2
Is digital cable television using IP? I thought it would be something
proprietary.

~~~
zbobet2012
IP to the edge (of a CMTS) and then it's flipped to QAM.

[https://en.wikipedia.org/wiki/Quadrature_amplitude_modulatio...](https://en.wikipedia.org/wiki/Quadrature_amplitude_modulation)
[https://en.wikipedia.org/wiki/Cable_modem_termination_system](https://en.wikipedia.org/wiki/Cable_modem_termination_system)

Also systems like AT&T uverse (fiber to the home) use IP multicast.

------
antisthenes
I'd be much more interested to see how much data was transferred during
streaming The International 6.

~~~
r1ch
According to my data, the TI6 English stream had 16M viewer hours.
Unfortunately I don't know the breakdown of what bitrate everyone was watching
at, but assuming a mix of 50% Source quality and 50% High quality, that would
equate to about 25,145 TB if my math is correct.

------
oskarpearson
Note that there's a mismatch between the listed title and the content of the
article. The article refers to "400Pb (400000Tb)", not "455Tb".

This may be why there's such confusion about the numbers in the comments.

------
visarga
I don't watch TV so I didn't see any of it. Was it free to stream?

------
tempodox
And then, nobody can watch because the IOC files a copyright complaint.

------
eva1984
Didn't seem too big to me..It is like 32Tb/4TB per day. What is the number of
Twitch's daily traffic, I'd say it is comparable.

------
user5994461
Let's say that an average video is 500MB nowadays. Be it a show, a short
movie, a long HD music video on youtube, or a sport event on the olympics.

Let's say that 1000 thousand people are watching the Olympic games. 500 GB *
1000 people = That's 500 GB of traffic.

Let's convert GB to Gb (multiply by 8) and we're talking 400 Tb.

So.. The title of this article means there are about 1000 people who watched
ONLY one event of the olympics and then put off their TV.

Well, I thought Olympics were a big worldwide event... I was wrong.

\---

Alternative: We can take it the other way around and say that there were 1000
sport events during the Olympic, and one guy to watch them all :D

Just for Fun: I'd love to see the numbers from Dailymotion.com, see how much
the Olympic Games are unpopular in comparison.

~~~
imperialdrive
agreed, this really isn't that much bandwidth

~~~
zbobet2012
2.0Mbps is an SD profile, which is worth mentioning.

