

Realtime encoding - over 150x faster - felixge
http://transloadit.com/blog/2010/12/realtime-encoding-over-150x-faster

======
JonnieCache
Maybe, if we keep bringing it up on here, startups will finally start putting
a couple of lines of boilerplate at the top of their blog pages so we will
know what on earth it is that they do when we end up there from HN.

A guy can dream, can't he?

At least this time the logo at the top links back to the main product page.
Oh, and I see that the title text of the logo has just such a description!
You're almost there guys!

~~~
felixge
Thanks for the suggestion, we definitely screwed that up.

------
aquark
I don't really get the 150x bit:

"Since our servers can encode video much faster than most of your users can
upload it, this means there is literally no more delay between the end of the
upload and the video finishing encoding. In the screencast above this makes a
150x speed difference."

Surely the upper bound is 2x if you could transcode faster than the upload
before.

Unless you are just measuring the time between the upload finishing and the
transcode being done. But why would a user care about that metric rather than
the total elapsed time?

~~~
felixge
> Unless you are just measuring the time between the upload finishing and the
> transcode being done.

That is what we are measuring indeed.

> But why would a user care about that metric rather than the total elapsed
> time?

Because that is the time where the user feels he has already done his part,
but the site he is on is taking forever to do what he needs it to do.

Another reason why we measure the time from upload done to encoding done is
because that's what our customers pay us to do.

~~~
sophacles
I demand you address the OP question of why you haven't fixed the physical
constraints of crappy upload bandwidth? Seriously, wtf would we pay your for
otherwise? </snark>

~~~
felixge
As funny as this sounds, fixing the upload bandwidth problem is also something
we will work on. Getting a good route between your users and servers can make
a real difference.

So at some point in the future we'll offer upload servers in all major
geographic areas.

~~~
nitrogen
It's interesting how difficult this can be. I was recently evaluating Linode
locations for a new virtual server, and though I'm geographically much closer
to California, the Texas location gave me almost 20x the download speed. To be
precise, I was able to download from a California node at 300K/s, while I got
close to 6MB/s from Texas.

Comcast used to route my traffic to California through Seattle, Washington,
then down to San Jose and then Fremont, but now, it's going to Texas first,
then across to San Diego, then up to Fremont, across a saturated link.

------
dkubb
It's nice to see people thinking about processing input as a stream rather
than waiting for the entire message to be received before doing anything.

If you start to think about input and output in web app as streams rather than
buffered data, alot of neat possibilities arise for reducing latency.

~~~
jon_dahl
This is often true, but not always - some video formats put header data at the
end of the file, not the beginning, so you can't just start encoding as bits
come in. Or if you can, you're encoding blind.

~~~
felixge
Sure, but we think it's possible to "prepare" those videos on the client site
before uploading.

------
GeneralMaximus
A video starts playing the moment you navigate to that page. Note to OP:
please add a play button to your video. The video should start only if the
user has specifically requested it. It took me a minute to hunt down the tab
that was the source of noise in my otherwise quiet work environment and, once
I found it, I killed the tab without even glancing at the page title.

------
felixge
As you can probably tell, we are super excited about this, feel free to ask me
anything : ).

~~~
zdw
This is really neat, but since you said "anything"... I have a legal question
that I haven't been able to find the answer for:

How do you deal with the licensing issues regarding open source encoding
software? Do you pay the MPEG-LA a fee directly for use of the software? Is it
per file/per minute/flat fee?

Just wondering about the mechanics of this, mainly for an in-house streaming
application.

~~~
jon_dahl
IANAL, but generally, you need a license if you're using open-source encoders
or decoders and you're the one who actually compiles them. (Again, not a
lawyer, but that changes ffmpeg from "a source code description of an encoder"
to "an encoder.")

MPEG video stuff is generally pretty cheap or free for low volume. Audio
codecs are fairly expensive, on the other hand. AAC + MP3 + AMR is >$20K in
minimum fees.

------
tstrong
One small question: if a user has their upload stream redirected to your
service, and for some reason this upload is unable to finish, are we now
forced to have the user try the upload again? It seems one advantage of the
two step process would be the ability to try the process again on behalf of
the user rather than making them wait and that should be weighed into the
convenience formula.

I've never used your service so I'm not sure exactly how the upload stream is
redirected to your platform, so this concern might not be totally valid if the
upload is running through the client platform anyways.

~~~
felixge
> if a user has their upload stream redirected to your service, and for some
> reason this upload is unable to finish, are we now forced to have the user
> try the upload again?

If the upload doesn't finish the user needs to redo it.

> It seems one advantage of the two step process would be the ability to try
> the process again

You got me confused here. If the upload never finished, there is nothing you
can do to fix this.

That being said, resumable file uploading is the next thing we'll tackle.

~~~
tstrong
> It seems one advantage of the two step process would be the ability to try
> the process again

I should have been more specific: consider the special case that the encoder
on your side encounters a random error whereas just a "plain" upload to the
client platform would have finished, so precluding any network transmission
errors.

Although, with resumable uploading and a simple go-between service on the
client platform between the user and your platform all manner of fanciness
could be achieved, I think. Thanks for quick response!

~~~
jemfinch
Your question is answered in the link itself. Did you read it?

------
matwiemann
The biggest challenge to me seems to be spinning up instances once you get a
spike of realtime jobs in parallel. Keeping the 'realtime' promise is often
hard once your systems go into production.

~~~
felixge
Of course. But since most people upload with insanely slow speeds, we can
actually handle a spike much better with this feature than we could before :
).

------
cnanon
Isn't the whole purpose of pre-encoding before uploading is so that you shrink
the file size and upload faster?

Why would I want to upload 4GB of video when I can encode it down to 700MB
then upload?

~~~
ItsBilly
It's not for end-users. It's a service for developers to add to their web
application. You can certainly try to coax all your users into pre-encoding
their video - and understanding how to do that - if you like....

------
csytan
Kudos on launching the new feature!

The pricing is a tad expensive, but doesn't look bad at all when you consider
the need for an on demand encoder for the entire time that the user is
uploading.

~~~
felixge
We know we're a little expensive. But we did this so we could lower pricing in
the future (which we will), and to attract people who are getting more value
out of our service initially.

If you have a project and pricing is a deal breaker, just email us and we'll
set you up with a discount.

------
damienkatz
I was thinking this should be done in realtime on the client machine and
streamed up the server. Then you cut down on both transfer and encoding time,
and use far less server resources.

~~~
felixge
This means shipping a plugin, or porting ffmpeg and friends to flash /
javascript. Maybe something to think about if we ever decide to raise VC : ).

~~~
detst
Isn't this a great use case for Native Client?

Maybe it's not ready but it seems to be something to look forward to.

------
jey
" _While this sounds easy in theory, it is rather difficult to pull off on
most stacks._ "

Why is it difficult on most stacks? Because it's tying up a request handling
thread?

~~~
felixge
Because most stacks don't have a single threaded, non-blocking I/O event loop.

Sure you can do this with threads, but it's gonna get very tricky to pump data
between a socket, file and process using threaded programming.

~~~
jey
First, I agree that it'd be silly not to use thread-per-CPU non-blocking I/O.

However, I don't see why it'd be tricky in a threaded server. The thread gets
an fd to recv() the incoming data from, and it popen()s an ffmpeg process then
loops to recv() and write() the data until done.

~~~
felixge
I'm not saying it can't be done. In fact the multipart parser we have build
for transloadit has already been ported to C++ [1] and I imagine Java has
decent libraries as well.

But most people just sit on a request/response oriented Python/Ruby/PHP stack
with possibly stupid buffering load balancing in between (nginx buffers
uploads).

If you build this from ground up, it is certainly possible with a lot of
technologies.

[1] <https://github.com/FooBarWidget/multipart-parser>

------
nitrogen
Is that "contact us" link correct? It adds an e-mail address to the end of the
URL, which conveniently opens a comment page, but it still seems wrong.

~~~
felixge
Ouch, fixed it. Thanks for letting me know!

------
jared314
It is not faster, just parallel processing with the upload buffer. This would
be a nice feature to add to most web servers/web scripting environments.

~~~
felixge
The fact that it's not actually faster is the entire beauty of it. While our
competitors have been super busy to actually make their encoding faster, we
have just taken a huge bite of free launch by making the encoding happen in
parallel.

------
falsestprophet
You may want to avoid using the word "sucks" in a professional context because
there is a population of people for whom the word evokes the idea of oral sex.
Try substituting "is not so good." This will have the added virtue of being
super charming in your German accent.

edit: I usually don't bitch about being downmodded. But don't you guys know
any old people? And know that old people tend to be in charge of things? In
any event, you shouldn't rely upon business leaders of any age being ignorant
of the language.

-<http://www.thefreedictionary.com/sucks> 5\. Vulgar Slang To perform fellatio on.

-suck, Old English sucan, corresponding to Latine sugere "to suck." It's of imitative origin. Meaning "do fellatio" is first recorded 1928.

-Slang sense of "be contemptible" first attested 1971

~~~
jonursenbach
Since when has the word "sucks" evoked that?

~~~
StrawberryFrog
Since before it hung out with the hip words.

