

Technical Overview Of VP8, An Open Source Video Codec For The Web - Anon84
http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/pubs/archive/37073.pdf

======
DarkShikari
This paper is staggeringly awful. Beyond the misleading stuff in the actual
meat of the paper, even just the results at the end seem to be completely made
up!

I attempted to replicate the results and found the following to be true:

1\. They used a different keyframe interval for the two encoders.

2\. They cherry-picked four test clips out of the hundreds available to find
the ones where libvpx did best. They did not show results for _any_ of the
most widely used test clips, as libvpx loses badly on all of them.

3\. The numbers on their graphs are simply wrong. Running libvpx with their
exact settings, on the exact same source, gives almost _1db_ worse PSNR than
they reported (note: I only tested Deadline CIF). The x264 results match.
Either the parameters they used don't match the ones they actually listed, or
the numbers are fudged.

The worst thing about all this? According to VP8 developers, this paper was
_already accepted_ into a major journal.

(If anyone's interested, I can post more detail on a few of the other things
wrong with this paper, such as how they compared a known-to-be-fast VP8
decoder with a known-to-be-slow H.264 decoder, or how they fudged the scale on
their graphs...)

~~~
haberman
Those are some pretty serious accusations without a whole lot of evidence or
specifics to back them up.

You are claiming the paper's authors lied about their numbers and "show
absolutely no interest in correcting even the obvious errors." Have you
actually contacted them to be sure you didn't make a mistake? If so, what did
they say? I find it hard to believe that they would respond to an accusation
of fudging their numbers by saying they don't care.

You also have a financial interest in the continued dominance of h.264 due to
the commercial licensing program for x.264
([http://mailman.videolan.org/pipermail/x264-devel/2010-July/0...](http://mailman.videolan.org/pipermail/x264-devel/2010-July/007508.html)).
This is a conflict of interest that is relevant to your opinion about other
video formats.

You also have a history of making unfounded accusations. "Well, this time, it
looks like [Tandberg] ran out of ideas, so they had to go use the cheat sheet:
open source." (<http://x264.nl/developers/Dark_Shikari/tandberg.html>) Later:
"Update: Tandberg claims they came up with the algorithm independently: to be
fair, I can actually believe this to some extent."
(<http://x264dev.multimedia.cx/archives/589>)

There's no doubt that you are an expert in video encoders. That's not a
license to make unsubstantiated accusations about them.

~~~
DarkShikari
_You are claiming the paper's authors lied about their numbers and "show
absolutely no interest in correcting even the obvious errors." Have you
actually contacted them to be sure you didn't make a mistake? If so, what did
they say? I find it hard to believe that they would respond to an accusation
of fudging their numbers by saying they don't care._

I've been on IRC with them since I made that post. I have changed that part in
the post though -- as even though they _appear_ to have shown no interest, I
have received information from an off-the-record source that they are actually
looking into it.

 _This is a conflict of interest that is relevant to your opinion about other
video formats._

We're developing xvp8 too, which we will also license. Personally, I don't
care who wins, as long as the encoders are good. xvp8 has the bonus benefit
that MPEG-LA doesn't demand license fees on it, so we can aim it at some
markets we wouldn't otherwise be able to target.

Now, yes, this does create an incentive for me to want to dislike _libvpx_ ,
as they are now a competitor.

 _You also have a history of making unfounded accusations._

Is filing for a patent that covers existing open source software suddenly "not
bad"? The accusation seems pretty founded to me! The only reason I retracted
any of it was because people _SENT DEATH THREATS_ to the programmer whose name
was on the patent.

 _Those are some pretty serious accusations without a whole lot of evidence or
specifics to back them up._

Specifics:

Video tested: Deadline CIF

Bitrate: 700kbps

libvpx version: latest git

x264 version: latest git

PSNR achieved by x264 with their settings: 44.507db (roughly matches the
graph).

PSNR achieved by libvpx with the settings they referred to: 43.625db
(definitely doesn't match the graph)

There are a few possibilities here:

1) libvpx has gotten about ~0.8db worse since they did the test, since I don't
have the revision number. Unlikely.

2) The settings they referred to aren't the actual settings they used. Highly
likely -- but even enabling the ARNR options, as they mentioned on IRC, only
brought libvpx up to about 44.0db.

3) They tweaked the settings specifically for the test. Possible, and not
quite as bad as literally making up numbers, but still cheating.

4) They made up the numbers.

But even if I was completely wrong on this, cherry-picking results in such an
extreme way is something you do in fluff marketing pieces, not in _an academic
paper_. The entire paper, from top to bottom, reads like marketing: it doesn't
mention a single downside of any of their features.

~~~
haberman
_We're developing xvp8 too, which we will also license._

But unlike h.264 (AFAIK), VP8 has a BSD-licensed encoder available. Even if
you beat libvpx in speed and/or quality, libvpx will probably be "good enough"
for most people. I'm guessing your demand for xvp8 will be much lower, and
that you won't be able to price it as high as x.264.

 _Is filing for a patent that covers existing open source software suddenly
"not bad"?_

That's not the accusation I cited. The accusation I cited was that they copied
from h.264, which you later admitted might not be true.

 _There are a few possibilities here:_

5\. They are using code that they have not yet publicly released.

6\. They made an honest mistake.

7\. They measure PSNR using a different tool, and the two tools have different
measurements for some reason (I don't have enough expertise in this space to
know if this is actually a possibility or not).

There are lots of possibilities. That's why you should really get an
explanation from them before you publicly call them liars.

~~~
DarkShikari
_Even if you beat libvpx in speed and/or quality, libvpx will probably be
"good enough" for most people._

Unlikely: a large percentage of the applications that license x264 are live-
streaming apps of some sort (videoconferencing, broadcast, app streaming, etc)
-- being slow is a deal-killer for these.

But you are right that this does change things somewhat -- though I actually
haven't thought too much about that.

 _5\. They are using code that they have not yet publicly released._

Likely false. According to my info so far, the tests were done using a daily
autobuild from their own system (which, IIRC, builds from git).

 _7\. They measure PSNR using a different tool, and the two tools have
different measurements for some reason (I don't have enough expertise in this
space to know if this is actually a possibility or not)._

There are two ways to measure PSNR: Overall/Global and Average. Both showed
similar results here though, so that's unlikely to be the issues.

 _6\. They made an honest mistake._

Could be. If so, I hope they fix it. I toned down the original post, and in
light of your comments, I've delayed making a blog post on the topic for at
least a few days until I know exactly what's going on. Thanks for the advice.

~~~
DarkShikari
And the end results. The numbers in the paper are correct. however:

1) The settings in the paper were lies. This will supposedly be fixed. This
was supposedly unintentional.

2) The clips are indeed cherry-picked (libvpx loses by ~1db on foreman with
the same settings). This may or may be fixed. This was intentional.

All other criticisms of the paper are valid.

~~~
haberman
Cool, thanks for getting all the facts. Though I'm not sure "lies" is the
right word for something that was supposedly unintentional.

------
dmaz
"...the vast majority of processors in the coming decade will have no more
than eight cores"

I hadn't heard this figure before, but it sounds like another "640K should be
enough". Although I do recall Intel predicting mainstream 12-core processors
by 2008.

------
johngunderman
I've messed around with VP8 a bit, and I'm very impressed with its
quality/compression ratio. However, when I last used it (August), it had
significantly slower transcoding times than any other popular codec. Hopefully
they've fixed it by now, because in all other aspects, VP8 is the best codec
I've used.

~~~
Scaevolus
Does VP8 have any advantages over H264 other than claiming to be patent-free?

~~~
astrange
No. And the official encoder is tuned for PSNR, so it looks worse than even
Theora.

