
A Future Without MPEG - Daemon404
https://blog.chiariglione.org/a-future-without-mpeg/
======
dwheeler
Put another way: The video patent cartels have fragmented due to greed, and so
the cartel enablers at ISO have lost their purpose. In addition, organizations
are tired of paying bridge tolls.

So we can finally have open standards for video that anyone can just use and
implement. AV1 in particular seems to be on its way to replacing them:
[https://research.mozilla.org/av1-media-
codecs/](https://research.mozilla.org/av1-media-codecs/)

~~~
makapuf
I guess it wont mean that parents on video encoding won't exist, rather that
now they wont be standardized. Not everyone can be sponsored by advertising
monopoly, some might need their technology to bring revenue by itself. However
yes, the pool broke by too greedy companies (IP based companies, not
technology ones) and hevc is laughable. See it explained by mpeg chairman
itself [https://blog.chiariglione.org/a-crisis-the-causes-and-a-
solu...](https://blog.chiariglione.org/a-crisis-the-causes-and-a-solution/)

~~~
dwheeler
They will probably be standardized, but not by ISO. Currently it looks like
AOMedia will take over, in part because unlike ISO they are willing to commit
to having an open standard unencumbered by patents.

------
WhatIsDukkha
This post is so "inside baseball" that it's difficult to discern what his
point or goal is?

~~~
fmajid
Chariglione was the charismatic leader behind the MPEG standards, including
those behind DVD (MPEG-2). The biggest challenge wasn’t technical but
political: getting a bunch of companies to agree to pool their patents so
licensing them could be streamlined.

For next-generation video beyond H.264 (AVC) and H.265 (HEVC), that consensus
has broken-down and his prediction seems to be that open-source standards like
Alliance for Open Media behind the AV1 video codec will prevail. It’s worth
noting that H.264 and H.265 are telecom videoconferencing standards defined by
the ITU-T, not consumer electronics standards defined by ISO, so I’d say the
loss of relevance of MPEG happened 20 years ago, even if MPEG (part of ISO)
was involved in both standards.

~~~
SahAssar
The point of AV1 is that it is royalty-free, not that it is open source,
right? x265 for example is a open source implementation of HVEC among others.

~~~
fmajid
You're right, of course, the two notions are orthogonal, but in practice an
open-source project like Mozilla cannot use royalty-encumbered codecs.

As for the royalty-free nature of AV1, it will only be known once all the
patent lawsuits are litigated. The AOM patent litigation fund in itself is no
guarantee, given the capricious nature of common-law courts.

~~~
SahAssar
I think the specifics matter here and I'm not in any way versed well enough in
law to know so correct me if if I'm wrong but:

Mozilla could absolutely use and distribute x265 but that would also require
them to have a license for the patents required for HVEC and would also
require the users to have a license (which is of course unrealistic).

The only reason firefox can play h264 is because Cisco paid for an unlimited
license with subdistribution and assigned it to mozilla.

So as h265/HVEC is concerned the problem for mozilla is that even if someone
wanted to pay the bill like cisco did they would not be able to get a straight
answer who and what they needed to pay for a license, right?

~~~
danbmil99
That's exactly right. While it is impossible to absolutely prove a negative
(that no one will make a credible patent claim against AV1) -- it's worth
noting that the line of codecs starting with VP3 that led up to AV1 go back to
research done in the early 90's, and there has never been a serious patent
claim against any of them.

This is at least partly by design: On2 had an explicit mission to create
competitive codecs that avoided conflict with the MPEG patents. I'm pretty
sure this was a major factor in their appeal to Google.

(Disclaimer: I was CTO &/or CEO of On2 up until 2003)

------
danbmil99
This is really a story of Open Source and shared intellectual property finally
winning out over expensive, proprietary standards that tried to present
themselves as something else.

Disclaimer: I was founder of on2 Technologies, acquired by Google and the
basis for av1.

There really Is quite a bit to this story, starting in the early 90s and
leading up to the situation today.

~~~
thrilleratplay
I used to work in the same building as on2 in Clifton Park. You could still
make out the company name on the sign next to the main entrance. Many people
are amazed to find out that the technology behind a lot of internet video has
roots in this area. The building also housed many of the developers for
company called Kitware, which are probably best known for cmake.

~~~
danbmil99
I remember that building well!

Our team in Clifton Park was the engine that built the VPx line of codecs.
Many of the engineers who worked there are now at Google in Mountain View.

~~~
thrilleratplay
I always wondered happened to the people after Google purchased on2. It would
make sense that Google would want to keep that knowledge and not just buy it
for the patents and source. Thank you for clearing that up.

------
zerocrates
I don't really understand what the point of EVC is. From my understanding it's
really _two_ standards, one royalty-free that's slightly _worse_ than
HEVC/H.265, and one patent-encumbered/licensed that's slightly _better_ than
H.265.

The post here says "EVC is promising because it provides a quality that is
comparable with or better than AV1, although less than VVC. EVC may have a
chance if a licence will be published. However, this has not happened yet." I
can only assume, from those numbers above, that "comparable with or better
than AV1" applies only to the encumbered/enhanced variant.

It's hard to see why anyone would bother to implement the "base" standard vs.
the already widely-deployed AVC/H.264, nor the "enhanced" one which seems to
be roughly comparable to AV1 but with licensing costs attached (and as the
post points out, no certainty at all about what that licensing would actually
entail). Apart from those companies that hold the relevant patents, of course.

~~~
ksec
The widely deployed AVC is still AVC Main Profile, which is what Youtube and
many other Streaming / Broadcasting uses due to devices compatibility. EVC
Baseline is expected ( or claims ) to be 20% better than AVC _High Profile_.
So you should expect ~30% better than current AVC all while being royalty
free.

The beauty of EVC baseline is that it is quite power efficient. Considering
AVC already require the least computation in modern codec, EVC baseline offer
30% reduction in bitrate while requiring NO increase in decoding complexity
and is actually 40 to 50% _less_ in encoding complexity.

Note: None of these has been tested outside of its members as it does not
provide a reference encoder due to "new" patents arrangement with this codec.
So we have no way to verify those claims.

Edit: Will add the link to document with those claimed figures later.

Edit2:
[https://www.youtube.com/watch?v=0Itt0cOvgXU](https://www.youtube.com/watch?v=0Itt0cOvgXU)

~~~
zerocrates
My understanding is that YouTube uses at least some High Profile, and only
Main for the lowest resolutions (though who knows how up to date that
information is), along with of course VP9 and AV1 where they figure they can.

Anyway, if those claims bear out then it would seem like EVC has a possible
future as an "AVC killer," though the computation complexity factors are
complicated by the wide deployment of hardware acceleration for AVC
encode/decode. Of course, hardware support could materialize for EVC as well,
but it does rather complicate possible adoption for what we'll call a "sub-
generational" improvement, in a world coming around right about now to AV1
hardware support.

Perhaps reduced computational demands could lead to a niche of usage in
unaccelerated environments. I don't know if there's an example of significant
uptake for an "in-between" step like this that one could point to as a
possible model for success, though.

~~~
drmpeg
A quick check shows 360p = Baseline, 720p = Main, 1080p = High.

------
rasz
Meanwhile China said fuck this shit and developed[1] own standards,
AVS/AVS+/AVS2, and is already running 4K broadcast tests.

[https://goughlui.com/2020/05/17/c-band-adventures-looking-
fo...](https://goughlui.com/2020/05/17/c-band-adventures-looking-for-
avs-4k-uhd-services-lte-5g-interference/)

[https://goughlui.com/2020/05/21/satellite-more-avs-
avs2-head...](https://goughlui.com/2020/05/21/satellite-more-avs-
avs2-headaches-test-4k-uhd-transmissions/)

[1] Seemingly no source code available for AVS2 decoders, dubious claims of
beating HEVC without independent tests, and China history of "independent"
development by copy&paste notwithstanding

~~~
ClumsyPilot
I would say that in this case, that's a reasonable decision.

If a physicist can't patent superconductivity (you can't patent discoveries
and laws of nature), should it be possible to patent compression algorithms?
The case for patents in software, as additional protection on top on
copyright, is not clear to me.

Frankly neither copyright nor the patent law is actually suitable for
software. Software deserves a tailor-made IP law that actually recognises the
difference between compiled code and source code, the rights of the consumer
to modify his device and keep it updated, etc. There should be an escrow-type
system, where protection is granted if code is hosted in a public repository,
and is made public after 20 years or so.

All executable should be treated in a standard way, and there should be no
EULA outside of enterprise systems. A consumer person does not need to read
EULA when they buy a book or a car, why should they do so when they buy a PC?

Until this happens, I feel developing nations shouldn't even recognise
software patents - copyright is protection enough.

~~~
afiori
> should it be possible to patent compression algorithms?

I would say yes, at least for lossy compression, where the challenge is not
about shuffling bits but finding semi-artistic graphical approximations of
common video patterns.

I also approve of an industry-wide effort to patent-unencumbered solutions.

------
makapuf
Am I reading it right ? The text says: > But there is a big news: MPEG passed
away on 2020/06/02T16:30 CEST. Does that really mean MPEG is no more ? Or is
it hyperbole about something that happened 2nd of june ? This would be huge
(and really sad).

~~~
TD-Linux
That is the date of an ISO resolution to split MPEG into several separate
working groups, a political decision (which doesn't disband MPEG in any way).
A bit more detail:
[https://twitter.com/enginetankard/status/1269409381495365632](https://twitter.com/enginetankard/status/1269409381495365632)

------
ksec
I am not sure where to begin. But let's start with EVC.

EVC Baseline is expected to be 20% better than AVC _High Profile_ , and EVC
Main Profile is expected to be 30% better than HEVC. The Codec is backed by
Huawei, Samsung and Qualcomm. If you are unfamiliar with the Smartphone
market, The three would represent roughly 80% of marketshare excluding Apple.

Now on to VVC. I am surprised by the sentence

 _MC-IF has 31 members, 7 of which are licensing entities (i.e. a little less
than ¼ of all members). The “industry” members account for just ½ of the HEVC
patent holders. It is hardly believable that VVC will fare better than HEVC.
It could very well fare worse because VVC adoption in broadcasting will take
years._

Well Yes. That is because 7 of those licensing entities already covered most
if not all of the industry. ( Excluding a few Open Media Alliance Member of
course ) The MC-IF actually includes _ALL_ of the current HEVC patents
holders. That is HEVC Advance and Velos Media, which is basically Samsung and
Qualcomm, along with many others that were not in any HEVC patent pool due to
disagreement in the first place.

( And If you notice the removal of the infamous Technicolor, they sold their
patents to another entity that is inside MC-IF, but I cant remember which one
on top of my head )

VVC is expected to be 50% more efficient than HEVC. And judging from its
reference encoder, this is the first time since AVC / H.264 era a video codec
that might actually live up to its claims. ( Normally marketing likes to use
unrealistic claims ) It really is the state of the art Video Codec, at a
decoding complexity that is quite manageable. ( Lower than AV1 )

So what does all that means? Samsung, Huawei and Qualcomm are also inside MC-
IF ( represented by different groups ). My guess is that EVC is basically a
backup plan or a gesture to MC-IF, if the licensing deal can be agreed upon,
they will go with EVC.

As a video codec enthusiast, I am extremely excited with both VVC and EVC.

As to MPEG ( Not to be confused with MPEG-LA ), I am not quite sure why he
said it is dead. I reread the article a few times and still dont quite
understand it. May be I am missing some context?

~~~
unlord
> It really is the state of the art Video Codec, at a decoding complexity that
> is quite manageable. ( Lower than AV1 )

I am afraid this has not been substantiated by any of the public decoder
demonstrations I've seen. Please see the most recent VVC technical update
presented at the MC-IF meeting on March 2nd of this year:

[https://a7dce6fd-e8f0-45f7-b0b0-255c5c9a28e1.filesusr.com/ug...](https://a7dce6fd-e8f0-45f7-b0b0-255c5c9a28e1.filesusr.com/ugd/0c1418_c8134202e0e845e290a9917ce38d8c60.pdf)

On slide 10 is a graph of VVC performance showing the VTM (VVC) decoder at
2.0x the complexity of HM (HEVC). On slide 12, the Ittiam production decoder
boasts 1920x1080 @ 24fps on a 4-core Cortex A75 @ 2.5GHz.

Compare that with this recent study of dav1d (AV1) decoder complexity on a
broad set of mobile SOCs, where 1920x1080 @ 24fps was easily reached by a
Google Pixel 1 from 2016! Using just the two LITTLE cores! Even higher frame
rates were achieved with more modern devices:

[https://www.reddit.com/r/AV1/comments/gncplq/av1_multithread...](https://www.reddit.com/r/AV1/comments/gncplq/av1_multithreaded_decoder_comparison_2020may19/)

Full disclosure, I contribute to the dav1d project and performed this study.

~~~
ksec
Yes. But I think it should be noted the HM and VTM are reference encoder and
decoder, they are not meant to be used for production nor in any way
optimised.

But dav1d is a truly amazing pieces of work!

~~~
unlord
> But I think it should be noted the HM and VTM are reference encoder and
> decoder, they are not meant to be used for production nor in any way
> optimised.

Sure, but my comparison was to the Ittiam production decoder.

------
sitkack
This is absolutely amazing, I had no idea this would come so soon. I wonder
how much the vp% series of codecs played a part, or netflix/amazon/youtube
controlling both ends of the channel?

The next device to get converted to the web is the smart tv, there is no
reason it literally has be anything different then a webpage, same goes for
Roku.

------
cure
If it is true, then good bloody riddance. Nobody is going to miss those rent
seekers. The future belongs to open, unencumbered standards.

------
superkuh
>It should be no surprise that the HEVC standard has some use in broadcasting,
but its use on the web is estimated to be at 12%. If one considers that
broadcasting is a rich but declining market and video on the web is constantly
rising, one understands that ISO standards will be gradually relegated to a
more and more marginal market.

True enough. But at least in the USA all the satellite companies, and their
set top boxes, are being forced to switch to HEVC because their half of their
physical frequency spectrum was stolen by telcos. So there'll be substantial
amounts of people making hardware for quite a while. The web isn't everything
yet.

~~~
nolok
What does a frequency have to do with a video format? And how do you define
stolen?

~~~
superkuh
The amount of data you can transmit over a radio link is proportional to the
span of frequency you use. Your bandwidth. Satellite operators used to have
almost all of C band's frequency span. Changes were recently pushed through to
give half of this to mobile telcos. So now the satellite operators have to
push the same amount of video/media with half the physical resource to do it.
So they need to send half the data. The FCC has literally told the incumbent
sat users to switch to HEVC to accomplish this.

------
sabas123
Can somebody explain what it means to hold the IP or rights of any of these
standards?

~~~
tialaramex
In particular IP here ends up meaning patents. Patents are a particularly
egregious right because though they were created with a specific trade in mind
they were almost immediately perverted so as to keep only one side of the
trade.

So the idea of patents is, if Alice invents something really fucking amazing
she explains the invention, pays the government some money and they publicise
her explanation but she is entitled to control who uses the invention for a
period of time.

This is a trade because now Alice's invention is available to everybody, no
chance Alice loses interest and it's lost for a thousand years before somebody
else re-discovers it, but on the other hand Alice doesn't need to secure
investors and risk the invention being a failure, she can demand terms like
10% of the profits from anyone taking those risks.

But, it soon turned out you don't need a complete working invention. Alice
describes half an invention, and then nobody else can use it without the other
half, but even if they figure out the other half on their own they owe Alice!
Alice doesn't even need to invent the whole thing, she can just describe the
easy half, the government signs off and then when somebody smarter finishes it
Alice gets rich!

Today in most of the world people have found ways to argue that computer
software, which is a work of literature and thus protected by Copyright (which
has different problems we won't discuss here) can also count as an invention
and be patented.

So the _best_ outcome for an outfit like MPEG is that Bob invents a really
clever technique that turns moving images into much less data than before,
totally just as a single flash of inspiration somehow, and rather than keeping
it secret, or using just in one MSDOS program in 1989 and then never again he
_publishes_ it as a patentable invention, then MPEG incorporates it in a video
standard called, say, MPEG Bob. And maybe everybody in the world cheerfully
pays Bob $100 each for this amazing invention. Hooray.

But very quickly the problem with MPEG is that Charlie gets a government
patent for the invention of, say, dividing five into three equal integers and
then hires a very good lawyer. Charlie's lawyer says it doesn't matter that
this is nonsense and can't work, because $10M worth of lawyers say you owe
Charlie for using this invention in MPEG Bob.

In between these extremes there are lots of problems less intrusive than
Charlie. The Springfield Higher Institute of Technology gets a patent on an
idea you, an expert in the field, have been telling people about for years,
but you never wrote it down so you can't prove it. Did they really invent it,
or just hear it second hand? Either way, they demand $1 each, but they _are_ a
university so maybe that's good? Although it is billions of dollars, and it
was really your idea if it was anybody's...

Leonardo, the author of this piece and MPEG leader, is quite sure that patents
are necessary, mostly because of the Bobs in this world though I suspect he'd
be sympathetic to the Springfield Institute too - to him the existence of
Charlie is an annoyance that we should all try to find some way past rather
than a fatal flaw in the entire endeavour.

~~~
rasz
> Charlie gets a government patent for the invention of, say, dividing five
> into three equal integers

Hey, you just described S3TC patent! Funnily enough S3 ripped off Apple,
copying verbatim Hoffert work at Apple Advanced Technology Group and
consequent patents from 1990 (US5046119A) for Apple Video 'road pizza' codec,
except their patent added "for texture compression" at the end. The patent is
about dividing colorspace between two points by dividing colors by 3 :-) and
is directly coped from Apple. Gotta love how wikipedia calls direct copy a
"This mode operates similarly to mode 0xC0 of the original Apple Video codec".

Amazingly S3 even had the balls to try and sue Apple for S3TC royalties in
2010. Sadly Apple loves patents and didnt bother trying to invalidate, they
were content winning on technicality. S3TC patents are expired now.

------
Silhouette
TL;DR:

A predatory business model based primarily on large companies exploiting legal
technicalities to seek rent from innovators has been rendered obsolete because
the large companies couldn't get their act together and the innovators did a
better job anyway.

Efforts to co-ordinate those large companies in those exploits are probably
now doomed, as are the organisations behind them.

The future almost certainly belongs to open standards and the community,
though someone who has built their career around the old business model isn't
happy about it.

~~~
nullc
It's a bit too quick to call propritary video codecs dead.

The billion dollar plus a year residual stream can pay for a lot of efforts to
keep trying to refresh the stream.

The author of the post seems to hold the position that mpeg falling apart is
because many companies have been too aggressive. But the lesson that some are
going to walk away with it that they haven't been aggressive enough.

~~~
Silhouette
Proprietary video formats certainly aren't dead yet. MP4 is still the most
common format by a long way in web work, for example.

I think it's clear which way the water is flowing, though, and I don't think
any amount of distress on the part of the fading powers of yesteryear is going
to change that. There is too much incentive and too much technical expertise
on the other side, and that side also has enough big players with their own
legal teams to block attempts at obstruction by questionable legal challenge.

~~~
nullc
Sure. A lot of progress has been made.

But don't underestimate what a firehose of money can do, especially one in the
hands of which is effectively a dying industry. (A small one, for sure, but
one that was formerly extremely profitable)

The big players are can bring significant resources to bare to be sure, but
they're also the least committed fundamentally. The right extremely beneficial
licensing terms for say, HEVC, in exchange for dropping out of AOM would
probably be extremely uhh "business smart" for more than a few of those big
names.

One of the challenges in this space has always been that patent licensing for
existing codec has zero marginal cost. Vendors can respond to the threat of
open alternatives with licensing concessions. I think part of why we see the
system collapsing a bit now for video is because they've managed to thicket
themselves to the point where its become extremely politically difficult to
negotiate in that manner.

------
videoborat
i'll just leave that here: [https://blog.playtherightfuture.com/aoms-
av1-patents-arent-f...](https://blog.playtherightfuture.com/aoms-av1-patents-
arent-free-youre-just-not-paying-directly-for-them/)

