

FIFA 2014 World Cup live stream architecture - dreampeppers99
http://leandromoreira.com.br/2015/04/26/fifa-2014-world-cup-live-stream-architecture/

======
jsaxton86
Interesting post! The author mentions that the first step was to "receive this
input and to generate HLS output to a known folder". I have a couple questions
about this step:

1: Did they re-encode the input? Or just repackage it?

2: I'm unfamiliar with EvoStream, but if it is ingesting RTMP and outputting
HLS, why did Globo need to bother generating an HLS manifest? Couldn't they
just use the one EvoStream created?

~~~
jbochi
I am glad you liked. I am not the author of the blog post, but I was a member
of the team.

1: We just repackage it. 2: The problem is that EvoStream stores the manifests
and chunks locally and we needed high availability. That's why we use an
external data storage. We have had up to 30 simultaneous streams, with 7
bitrates each and 2 hours of DVR.

~~~
flavioribeiro
Just adding one extra information, building the playlist on our side enabled
us to define the DVR window (how many time one can seek back on the video).
Since we built both the server and client side, it was also possible to add
tags to the stream and we did it. tl;dr; It was possible to control the
adaptive streaming on the server side.

When our CDN was very crowded, one tag on the HLS playlist was able to direct
users to a lower quality, preventing all the users to fight for the same
bandwidth and avoiding rebuffering events.

------
kweks
Honestly - one of the most informative, well written, extremely well
referenced articles I've read literally in years. This article will teach me
more than I've learned in a long time. Thank you very much for your
contributions..

------
emilsedgh
Just like everyone else, I really liked the article. And of course I have some
questions.

What is the benefit of EvoStream over nginx-rtmp? I use nginx-rtmp as well and
it creates HLS files just fine.

Also, how did you handle the network load? How many edge servers were actually
delivering the content to users?

Did you use 10G ethernets, link bonding, or did you use cloud services (eg
cloudfront) to deliver the content?

Thanks a lot.

~~~
dreampeppers99
* We initially picked EMS because of its API, but we're moving to nginx-rtmp.

* We used 20G ethernet, we could get 19G from each machine.

* All the content is hosted by us.

~~~
emilsedgh
Thank you for your answer. Appreciate it and best wishes.

------
gregor7777
My favorite type of article, thanks! Even better when the stuff they're using
is open source so I can check it out for myself.

------
icaromedeiros
It is rewarding working at globo.com with such skilled engineers and great
solutions, in almost every area. Congrats to Leandro, Juarez, and everybody
involved.

------
alexkappa
Thanks for sharing this! How long was your team given to complete this
project?

    
    
      Cassandra response time was increasing with load to a certain point where clients started to timeout and the video playback completely stopped.
    

and

    
    
      After these changes, we were able to achieve a latency in the order of 10ms for our 99% percentile.
    

How did you test this? Are there any benchmarking/load testing tools for cases
such as streaming?

~~~
dreampeppers99
If I'm not wrong, we start to access that around 1y but I think we really
spent 7-10m on it.

You can send metrics from Cassandra to graphite
[http://www.datastax.com/dev/blog/pluggable-metrics-
reporting...](http://www.datastax.com/dev/blog/pluggable-metrics-reporting-in-
cassandra-2-0-2)

Since the streaming is similar to HTTP page flow, it's not that hard.

Ex: [http://blazemeter.com/blog/how-load-test-http-live-media-
str...](http://blazemeter.com/blog/how-load-test-http-live-media-streaming-
hls-jmeter)

------
Finster
It feels like they reinvented the wheel on a lot of things (especially on the
monitoring side). I also wish there was some more detail about the
caching/endpoint side of things. We rely heavily on Akamai's services for
their edge servers, caching, and site acceleration. I don't see anything like
that here, but maybe they're just not mentioning it?

~~~
dreampeppers99
We only implemented the dashboard (getting data from all kinds of source) on
monitoring side, we relied on: logtash, graphite, elasticsearch...

The architecture up from what was described is almost purely to provide
caching capabilities, aka bunch of edge servers caching content to final
users.

------
robinhowlett
Seems like quite a complex solution for 500,000 concurrent users.

Why not just use Akamai for RTMP ingest and delivery? From my experience they
can do all that was described and at scale.

~~~
dreampeppers99
Please keep in mind that FIFA14 was one event, we still broadcast many other
events and channels simultaneously ( private and public).

About why chose HLS over RTMP, in our experiments HLS showed to be easier to
scale than RTMP (maybe because the latter is stateful + the current players
don't have an optimized adaptive bitrate algorithm + it's easier to scale http
over rtmp)

~~~
robinhowlett
Yes agree, but Akamai can also transcode and repackage to HLS. We've used this
at SportsLabs[1] to stream tens of thousands of sports events.

[1] [http://sportslabs.com/](http://sportslabs.com/)

------
superted
Working in the TV industry myself, the uninevitable question on DRM springs to
mind. Was these streams encrypted as well somewhere along the described
process?

------
eder_roger
A great example of use the powerful and free tools. Congratulations, great
article.

------
wjsf
Great experience, thanks for sharing.

------
robertomb
Wonderful job, guys. Will you broadcast Olympics? What will change for this
one?

~~~
dreampeppers99
Yes, for sure. :) Probably, there will be some changes but we're going to
discovery some of them ongoing.

------
flavioribeiro
beautiful! congratulations for sharing this to the community.

------
aprdm
Really well written article! Learned a lot, also happy that it came from
Brazil :)

------
rarararararara
Here's all you need to know about this company (Globo):
[https://www.youtube.com/watch?v=77TKLQ1op34](https://www.youtube.com/watch?v=77TKLQ1op34)

This documentary is still banned in Brazil and can't be broadcast offline
anywhere.

~~~
outworlder
This documentary is very old and pre-dates the internet.

There's no denying the extent of Globo's influence. But, compared to the 80's,
it's merely a shadow of it's former self. Probably because of the internet.
Also, there are other communication vehicles fighting for eyeballs now.

Politics aside, its IT arm 'globo.com' is pretty modern and open.

~~~
personlurking
> There's no denying the extent of Globo's influence. But, compared to the
> 80's, it's merely a shadow of it's former self.

The ire lingers, though. Sometimes Globo shows up to report on something and
they essentially get kicked out of the area (by the people near the event).
This may be a combination of how it tends to report on certain subjects and
leftover distrust from the past decades.

