
Justin.tv's Live Video Broadcasting Architecture (2010) - kd5bjo
http://highscalability.com/blog/2010/3/16/justintvs-live-video-broadcasting-architecture.html
======
glacials
I worked at Twitch from 2014 to 2018. I was never on the video team, but here
are some details that have changed.

Video:

\- Everything is migrated from RTMP to HLS; RTMP couldn't scale across several
consumer platforms

\- This added massive delay (~30+s) early on, the video team has been crushing
it getting this back down to the now sub-2s

\- Flash is dead (now HTML5)

\- F5 storms ("flash crowds") are still a concern teams design around; e.g.
900k people hitting F5 after a stream blips offline due to the venue's
connection

\- afaik Usher is still alive and well, in much better health today

\- Most teams are on AWS now; video was the holdout for a while because they
needed specialized GPUs. EDIT: "This isn't quite right; it has more to do with
the tight coupling of the video system with the network (eg, all the peering
stuff described in the article)" -spenczar5

\- Realtime transcoding is a _really_ interesting architecture nowadays (I am
not qualified to explain it)

Web:

\- No more Ruby on Rails because no good way was found to scale it
organizationally; almost everything is now Go microservices back + React front

\- No more Twice

\- Data layer was split up to per-team; some use PostgreSQL, some DynamoDB,
etc.

\- Of course many more than 2 software teams now :P

\- Chat went through a major scaling overhaul during/after Twitch Plays
Pokemon. John Rizzo has a great talk about it here:
[https://www.twitch.tv/videos/92636123?t=03h13m46s](https://www.twitch.tv/videos/92636123?t=03h13m46s)

Twitch was a great place to spend 5 years at. Would do again.

~~~
diminoten
> \- No more Ruby on Rails because no good way was found to scale it
> organizationally; almost everything is now Go microservices back + React
> front

Ugh, I just... I keep trying to pretend I don't need to learn Go, but _every_
highly scalable system I read about that's recently been written about seems
to be using it. Maybe I just need to stay away from systems that need to
scale? Heh...

~~~
apta
Before golang was a thing, there were highly scalable systems that handled way
more traffic than anything written in golang today. Those systems were (and
are) written in languages like C++ and Java and C#.

You're just seeing golang in articles because of hype.

~~~
hactually
Java and C# lag behind Golang on most performance metrics. Combine it with the
awesome deployment story (single binary) and you'd be hard pressed to choose
the former

~~~
MrBuddyCasino
The opposite is true. Golang is on average 2-3 times slower than Java. On the
plus side, it uses less memory.

~~~
LOLOLOLO1
No of course. It may loses at some benchmarks made by Java or Python/Ruby
coders.

Two areas where it is actually slower are

1) Memory allocation 2) Regular expression performance.

But you must understand your Java app won’t have performance advantage because
of faster alloc speed IRL: GC will take lots of CPU, because Go’s one is much
easier at memory release. You can only see allocation advantage in micro
benchmarks where the app stops before the GC will start.

~~~
apta
> You can only see allocation advantage in micro benchmarks where the app
> stops before the GC will start.

The golang gc is tuned for latency at the expense of throughput, meaning if
you look at the duration of time spent in GC over the course of the code
execution, it would actually be longer compared to a GC tuned for throughput.

If you have a use case that requires high throughput, then you cannot change
the GC behavior. Unlike on the JVM, where you have several GCs to choose from.
The JVM is also getting two new low latency GCs for use cases that require low
latency.

And it's not just microbenchmarks where Java does better than golang, it's
especially longer running processes where the JVM's runtime optimizations
really kick in. Not to mention that the JVM is getting value types as well to
ease the load on the GC when required (it does an excellent job as it is even
without value types).

I did a dummy port of the C# version of the Havlak code here[1] to Java,
preserving the same behavior and not making any data structure changes. On the
machine I tested on, the C# version took over 70 seconds to run, while the
Java version took ~17 seconds. In comparison, the C++ version took ~24
seconds, and the golang version took ~30 seconds.

Yes, you could most likely spend much more time tuning the C++ version,
avoiding allocations, and so on, but at the expense of readability. This is
what the JVM gives you, you write straight-forward, readable code, and it does
a lot of optimizations for you.

The brainfuck2 benchmark is especially interesting. Kotlin tops the list, but
I was able to get Java to the same performance since Kotlin by writing it in a
similar manner as the Kotlin code. Again, Java/Kotlin beat out even C++ when I
tested them, and by quite a margin.

[1]
[https://github.com/kostya/benchmarks](https://github.com/kostya/benchmarks)

------
aerovistae
I can barely follow along with this, it's very technical. I can't imagine how
Kyle Vogt acquired the necessary knowledge to make this work. Example:

> The point of having multiple datacenters is not for redundancy, it's to be
> as close as possible to all the major peering exchanges. They picked the
> best locations in the country so they would have access to the largest
> number of peers.

This is the kind of thing where I would have to hire some kind of network
engineering expert, and he just _figured this stuff out_ and made it work? I
can't fathom other people's intelligence sometime.

~~~
jedberg
He leveraged the YCombinator network to absorb a lot of information quickly.
For example, I taught him basic networking (routers, switches,
multicast/anycast, AS numbers, etc). I shared my 10 years of knowledge with
him in a single two hour session because he's a genius, and then he ran from
there, vastly exceeding my knowledge. I was there because Emmet asked Steve
and Steve asked me to go over, and I was happy to help. I'm sure I wasn't the
only one.

~~~
nickpsecurity
That's pretty nice. Quick look at articles show Emmet was the CEO. Which Steve
is this? Huffman at Reddit?

~~~
jedberg
Yes Steve Huffman

------
kvogt
Building this was really fun, and I’m very proud of what kd5bjo, Emmett, and
many others did to help turn Justin.tv/Twitch into what it is today. We found
a way to make what was fundamentally an unprofitable business (if you relied
on CDNs) work by relentlessly focusing on reducing cost to the absolute bare
minimum through good technology choices and innovating when necessary.
Justin.tv would have died otherwise.

~~~
JoeOfTexas
Thanks for sharing the information! Even if its 9 years old now, its very
interesting for us dreamers.

------
dang
Discussed at the time:
[https://news.ycombinator.com/item?id=1196264](https://news.ycombinator.com/item?id=1196264)

------
kd5bjo
I was the primary architect for Usher and the server-side of the video system
described here. I’m happy to answer any questions, assuming I still remember
the answers.

~~~
loljjkm8
Why did you call it Usher and not T-pain

~~~
kd5bjo
I don’t get your reference, but we chose the name Usher because it’s the
software equivalent of the person at the theater who looks at your ticket and
shows you where your seat is— it doesn’t actually handle any of the video
data, it just knows where you should go to find it.

~~~
spydum
Surprised Microstrategy didn’t sue you guys (or vice versa). They built some
access / authorization tool named Usher which seemed to be a total flop (why a
BI company was even toying with this perplexes me)

~~~
kd5bjo
It was/is an internal code name that never shows up except in URLs requested
by the player and highly technical articles like this one; I’m not a lawyer,
but I expect they might have a hard time showing consumer confusion.

------
Nition
Slightly off topic but this is my favorite justin.tv video, and also shows
something of the website and how early in the days of live streaming we were
back then:
[https://www.youtube.com/watch?v=BqgEm8XWXu8](https://www.youtube.com/watch?v=BqgEm8XWXu8)

~~~
64738
Thanks for that! I hadn't seen it before, that's pretty funny.

------
joefourier
"Live video can't be made by pushing video faster, it takes a completely
differently architecture."

Funnily enough, that's pretty much how HLS, the modern live video standard,
works - it's essentially a series of tiny video clips loaded & played one
right after the other, distributed through the same CDN as normal video files.

Thanks to HLS, live video is actually much worse than it was 10 years ago with
RTMP in terms of latency. There's been some recent efforts in getting it down,
although they're generally not standardised, hard to scale (e.g. WebRTC) and
or a bit awkward.

~~~
spenczar5
They're starting to become standardized. Apple announced LHLS, a "low latency
extension" to the HLS standard,
[https://developer.apple.com/documentation/http_live_streamin...](https://developer.apple.com/documentation/http_live_streaming/protocol_extension_for_low-
latency_hls_preliminary_specification).

I don't think anyone really follows Apple's spec for various technical
reasons, though. Most do some sort of chunked-transfer encoding, along with
pre-signaling segments in playlists, as outlined by the Periscope folks here:
[https://medium.com/@periscopecode/introducing-lhls-media-
str...](https://medium.com/@periscopecode/introducing-lhls-media-streaming-
eb6212948bef)

~~~
eridius
> _I don 't think anyone really follows Apple's spec for various technical
> reasons, though._

I would have assumed it's because Apple only announced this 4 weeks ago, and
the only clients that support it are beta software.

------
SaltyBackendGuy
> They also don't have chats with the 100,000 people watching a channel. What
> they do is assign people into rooms of 200 people each so you can have a
> meaningful interaction in a smaller group. This also helps with scaling. I
> thought this was a pretty clever strategy.

Just curious if this is still a thing. I've watched an unhealthy amount of
Twitch (not all with chat open) and never noticed this.

~~~
savanaly
I'm sure it's not. Often, the streamer is reading the chat onscreen while
streaming, and it's always identical to the one I'm seeing, except perhaps
delayed by some seconds.

~~~
doomjunky
Maybe twitch creates an illusion by showing the streamer only a subset of
people that is shared among all other groups. This allows everyone to see what
the streamer sees and to communicate within their group.

------
augbog
Really cool reading this when I just started 2 weeks ago and I was perusing
the Video Architecture docs. It's cool to see how far Twitch has come :)
thanks for sharing

------
olah_1
If I started a Twitch (or Discord) competitor today, I would use Janus as a
base for web-rtc to share the load
[https://janus.conf.meetecho.com/](https://janus.conf.meetecho.com/)

In fact, Evasyst is a blend of Twitch and Discord and uses Janus for this
reason. [https://evasyst.com/](https://evasyst.com/)

~~~
slimscsi
You would look at how much that costs and make a different decision. It can’t
scale economically to Twitch size without massive amounts of investment into
POPs (source: I built the HLS transcoding stack/infrastructure and HTML5
player at Twitch). A voice only service, sure because it’s 100 times less
bandwidth.

~~~
kd5bjo
Granted, we made the same evaluation about Wowza a decade ago, but it turned
out to be useful as a small piece of a large system. Given how much Twitch has
grown since then, no off-the-shelf system would be able to handle the scale.

A brand-new competitor, though, won’t have to deal with those scaling problems
for a while. They have to figure out product-market fit first, which probably
can be done off-the-shelf these days. And then they’ll need to figure out how
to keep the lights on, which will take a similar amount of engineering work as
it did for Justin/Twitch.

------
imjustsaying
Can anyone recommend some stacks for hosting video streaming?

For 10k concurrent stream receivers in the US and Europe, what would you be
looking at to get the stream under 30 seconds lag time?

I realize this is a hard problem but would like to explore it.

------
simlevesque
Wowza is a great product. If you want to build a huge custom VOD + live
boradcasting product, it is a great way to do it.

------
foobar_
I used to think high scalability was something to aspire to. Now I think it's
all bullshit.

The web is a terrible architecture for videos. Torrents solved this problem.
If you trying to make a camel walk with two legs, you don't deserve praise but
a tomato.

------
person_of_color
How would an IC SWE find a rocket ship startup like this in 2019?

~~~
cbzehner
Check out [https://breakoutlist.com/](https://breakoutlist.com/), it's
designed for answering just that question - with a bias towards Silicon Valley
hype.

If you're interested in Atrium, reach out to me (see profile). We're hiring!

~~~
person_of_color
Awesome resource. Time to send some resumes over the weekend.

Who's with me?

------
OedipusRex
I miss the Justin.Tv days. Made a lot of friends there.

~~~
jrs95
Some parts of Twitch are still kind of like that. The Classic WoW community
especially has been amazing. Met a lot of people there that I play games &
just hang out in Discord with pretty much every day.

~~~
OedipusRex
Twitch spun out of Justin.tv and then they killed Justin.tv :(

