
Using Microservices to Encode and Publish Videos at The New York Times - kawera
http://open.blogs.nytimes.com/2016/11/01/using-microservices-to-encode-and-publish-videos-at-the-new-york-times/?_r=0&pagewanted=all
======
enobrev
I worked on something like this for a small video rendering startup a few
years ago. Our backend needed to ingest short (<= 12s) videos and strip the
parts we needed in our rendering stack. We polled S3 (SNS wasn't consistently
fast enough at the time) for new files in our Upload directory.

The uploaded video was sent to multiple microservices in a group of tens or
hundreds of instances. One would "normalize" the video so we had the same
frame-rate, size, and bitrates to work with. Another would rip the audio into
a couple formats. Another would generate HLS streams with a few resolutions.
Another would generate the HLS playlist. Another would split the video up into
images. Each service uploaded its results to S3, named as a SHA1 hash of
itself, and reported to a single API which kept track of the "tree" of files
generated from each upload.

The system was based upon an early version of NodeJS, which worked
fantastically well as a generic daemonized wrapper around the command line
tools we were using for all the ripping and modifying.

About a year before I left the team we changed to GCS for the upload server
because they had reliable immediate notifications to webhooks for uploads, but
kept the rest of the system on S3, since our processing instances were all
EC2.

Would have loved to have Lambda back then!

We had a similar set of microservices for rendering the eventual videos in
near-real-time - but that's a whole different conversation.

------
mattmanser
Seems a bit, um, over-engineered.

I made one of these 6 years ago for one of my clients using aws, on my todd
and it's really not that complicated. Sure ours is a bit duct-tape and string,
but seriously, it's not complicated programming. Probably took me a whole day.

At one point we switched from one encoding service to another and it took me
less than half a day of chargeable time to change the code, so not sure why
they need it to be so "flexible".

They've probably wasted more dev time on over-engineering it than they'd ever
have saved manually adding a new encoding provider.

How it takes 4 of them to write it too I'm a bit mind-boggled. Reliable
processing queues are a solved problem, encoding services are a solved
problem, this whole thing was a solved problem 5 years ago, which is aeons in
programming times.

------
marzagao
Hello all, my colleagues Flávio and Max also did a presentation of this system
at the Streaming Media West conference recently.

Here are the slides: [https://speakerdeck.com/flavioribeiro/building-a-video-
encod...](https://speakerdeck.com/flavioribeiro/building-a-video-encoding-
pipeline-at-the-new-york-times)

------
xchaotic
"The main goal of the team was to implement a job processing pipeline that was
vendor agnostic and cloud-based" I wonder if they feel they've achieved it?
How much work would it be to switch away from their choice of JS libraries and
AWS? And if it's not trivial, what's the point?

~~~
standel
IMO, they probably didn't want to be cloud vendor agnostic. They really wanted
to be /video encoding/ vendor agnostic.

~~~
xchaotic
Do you need 20 microservices for that? If you choose to swap a video vendor, I
guess you need to write wrapper API calls for the new vendor? Even if the new
vendor provides that out of the box, NYT amortizes those costs one way or the
other...

~~~
tveita
Where do you see 20 microservices? I count a couple of video-specific ones, +
some integrations with their existing systems.

~~~
xchaotic
It may be more or less, depending on how you count the, but what I'm really
trying to figure out is whether the over engineering and the extra abstraction
layers are justified in this case and in general. My experience so far is the
opposite - Microservices, orchestration etc hide the complexity but don't
remove it, so if the shit hits the fan, you still have to look under the hood
and with Microservices, probably under 10 different hoods or a hood under the
hood of a hood :)

