
AVIF for Next-Generation Image Coding - dedalus
https://netflixtechblog.com/avif-for-next-generation-image-coding-b1d75675fe4
======
roca
It's been a slow process, and hence perhaps underappreciated, but it's very
gratifying that over the last 15 years patent-unencumbered media codecs have
won. The companies that contributed to this --- Google, Mozilla, Cisco, and
others --- and especially Xiph that got the ball rolling --- deserve a lot of
credit.

~~~
greggman3
They have? Are we talking images only because in video mp4 and .h264 and .h265
are pretty much it. Apple/Safari still doesn't support vp8/vp9/webm/av1 for
video or ogg for audio

~~~
est31
> Are we talking images only because in video mp4 and .h264 and .h265 are
> pretty much it. Apple/Safari still doesn't support vp8/vp9/webm/av1 for
> video or ogg for audio

h265 is doomed because of the multiple patent pools that have formed and the
extreme price hikes compared to h264. It's a risky technology to build on.
That's the best ad that AV1 can have, and many members joined the alliance for
open media for that reason after it became clear that the patent pool
situation wouldn't resolve.

As for your browser support question, yes, Safari has traditionally been anti-
ogg, but a few years ago Apple joined the alliance for open media and added
opus support to their browsers (although only in the caf container, for some
weird unexplainable reason).

Open codecs have won in the lossless audio domain (flac). There is no reason
they can't win in the lossy video and lossy audio domains too (not sure about
lossless videos but FFV1 seems to have lots of support by archivists who want
open technology).

~~~
p0nce
> h265 is doomed

I strongly recommend to see this video:

"The state of advanced codecs; separating hype from reality"
[https://www.youtube.com/watch?v=vgE8-4rcXl0](https://www.youtube.com/watch?v=vgE8-4rcXl0)

~~~
akiselev
Care to summarize the relevance of that video to the topic at hand?

~~~
ksec
If your world of Video = Youtube. Then HEVC failed.

If your world of Video = Every Video usage on the planet Earth. Then HEVC
isn't doing so bad, every shipping TV has had HEVC decode, most ( if not all )
Smartphone, PC, Tablet has hardware Decoder shipped.

~~~
BurningCycles
>Then HEVC isn't doing so bad

Sure, but it was massively pushed as the successor of h264 as the new de facto
video codec standard, after almost a decade in use it has clearly failed in
that respect.

IMO this codec generation saw no winner, instead h264 remains the undisputed
king, the next battle for the crown will likely be between AV2 (or whatever it
ends up being called) and VVC (Versatile Video Coding).

~~~
webmobdev
> instead h264 remains the undisputed king

Sure, because it is "good enough" for what it does. But recently, with the
advent and demand of 2k and 4k videos, people realised the need of, and have
started appreciating the capabilities of HEVC.

When it comes to the popularity of video encoders, my unscientific way of
judging it is to look at what the pirates are using. If you look at the
torrent scene, most of the popular tv series and movies are now also available
HEVC encoded and they are popular too (especially the high quality 2k and 4k
Blu Ray rips). It's rare to come across any VP9 videos, and HEVC is
undoubtedly number 2.

Personally, even I've started re-encoding my video library with HEVC, once I
discovered that the "medium" setting takes about the same time as H.264
encoding and gives an acceptable quality for (sometimes) half the file size.

I do keep an eye on encoders, and while it is good to know the work going on
this field, practically speaking nearly everyone is moving on to HEVC. Nearly
all the other competitors are in development and unusable because of their
really slow encoding time.

~~~
p0nce
> practically speaking nearly everyone is moving on to HEVC. Nearly all the
> other competitors are in development and unusable because of their really
> slow encoding time.

Be careful when saying the truth!

~~~
webmobdev
Forget commercial softwares, even the popular open-source video encoders like
Handbrake or AviDemux or FFMpeg do not have AV1 or any other competing
encoders in their release. They do support HEVC though. That itself is telling
on the state of the competitors.

Note that I am in no way an advocate for HEVC (even if I sound like one :). I
am just speaking from a practical point of view, as a user. If tomorrow there
comes another encoder that takes less time to encode, and offers better
compression, I'll immediately dump H.264 and H.265 for it.

------
pornel
This is a major improvement. It's better than WebP by a bigger margin than
WebP was better than JPEG.

Lossy WebP couldn't even match JPEG in some aspects, like full-resolution
color (4:2:0 only, not even full 8-bit depth). In terms of quality or features
AVIF doesn't have such "buts". It can support HDR and high depth we're likely
to want in the near future. AV1 has palette _blocks_ , so it's no longer all-
or-nothing choice of lossy vs lossless formats.

I am worried that the AVIF spec is complex (like HEIF, it's a bag of image-
related features, more like a PowerPoint file than an image), but probably
only commonly-used features will survive out of it. Early JPEG also used to
have many more modes than are usable today.

If you like questionable hacks, you can use AV1 codec today in Firefox and
Chrome by embedding <video> element with a 1-frame AV1 "video".

~~~
ec109685
The JPEG images in the article are much worse than what you can achieve with
JPEG export in Preview.app. While they don't look like AVIF, they aren't
nearly as trash as the article makes JPEG's out to be.

------
reggieband
I worked on a team responsible for image assets. I recall the web team asking
for webp support from our image server. I suppose nowadays using `srcset` you
can get some optimization in the ratio between file size and quality while
still providing fallback for unsupported browsers. Given our image service was
dynamic (e.g. it generated then cached images at different sizes, qualities,
etc from a single high-res source) adding a new format wasn't that difficult.
It meant that the image heavy pages would display more quickly which was a
concern especially for mobile. So I am a big supporter for more and better
formats.

That being said, this AV1 vs. HEVC / AVIF vs. HEIF stuff feels eerily similar
to HLS vs. DASH. It's not like these formats have drastically different
properties (e.g. GIF vs. JPG vs. PNG) - they are all quite similar. I just
want to fast-forward time until whichever one is going to win is standard.
It's like VHS vs. Betamax or Blueray vs. HD-dvd. Please just get it over with.

~~~
MikusR
The difference is that HEVC/HEIF has at least three different patent licensing
organizations that all want to be paid. While AV1/AVIF is supposed to be
"patent free"

~~~
reggieband
I heard the same about h.264 vs VP9 and it turned out that patent licensing
wasn't a long-term issue. For a while you got that license from Flash (if
y'all remember when Flash was the way Youtube worked, and I recall mention
that it was one of the primary reasons Flash Player was never opened sourced).
Then the license for h.264 it was included in every copy of Windows 8 (IIRC).
Nowadays no one even mentions the h.264 license.

I'm no expert on the new license but it seems the restrictions on HEVC are
relatively light. I'm pretty sure it is free to compress and distribute media
but the license affects hardware and software encoders/decoders. So maybe your
HEVC enabled video card with built-in decoder will be a dollar more expensive.
I know that every Apple device shipped comes with HEVC hardware standard for
the last few years. I would wager most Android devices too.

~~~
izacus
No, HEVC licensing is an insane mash of several patent pools and a few
companies trying to extract a buck out of anyone that uses it. It's
significantly less clear and more complex than what H.264 was and the license
fees are higher as well... If you even have a legal team that can figure out
who needs to be paid.

I'm on the phone so I can't really look up all the resources, but they
shouldn't be hard to find. But this licensing mess is also why most oss
development stalled and we still don't have an encoder that would be anything
close to the quality of libx264. Instead, most important OSS experts have
moved on to AV1.

~~~
reggieband
> HEVC licensing is an insane mash of several patent pools

Yes, but they aren't coming after users. Look at the pools, it is mostly
device manufacturers like TV, audio systems and mobile phones. They are
holding each other in a mexican stand-off. Two of the three largest pools have
already agreed to the same $0 royalty for non physical distribution that h.264
has. They want to extract that money from their competitors in the physical
device space.

Fair enough, the uncertainty around final pricing is slowing adoption. But
that can ossify quickly, although no guarantee it will.

I hate I'm being drawn into a patent argument when my point is: it won't
matter either way. Even if HEVC wins 99% of people will never be affected by a
license fee. Let them scrap over $0.25 royalty fees on TVs all they want. In
the end I don't care if it is HEVC of AV1 that wins. I just want the fight to
be over so I don't have the cognitive overload of having to support both.

------
gok
That 420 vs. 440 comparison animation is rather problematic. GIF can only
represent 256 colors. Most of the artifacts in those images are due to GIF-
specific dithering.

also, previously:
[https://news.ycombinator.com/item?id=22327480](https://news.ycombinator.com/item?id=22327480)

~~~
Dylan16807
Should have used apng, sure, but the _difference_ is still very visible on
those hard color lines.

------
JyrkiAlakuijala
encode.su forum posts compare image quality of JPEG XL, AVIF, WebP, MozJPEG
and HEVC:

[https://encode.su/threads/3108-Google-s-compression-
proje%D1...](https://encode.su/threads/3108-Google-s-compression-
proje%D1%81ts/page3)

The most comprehensive comparison:

[https://medium.com/@scopeburst/mozjpeg-
comparison-44035c42ab...](https://medium.com/@scopeburst/mozjpeg-
comparison-44035c42abe8)

"... according to my personal visual tests, starting from 0.4–0.5 bpp it [JPEG
XL] usually wins most other formats."

------
oconnore
Shouldn't this be benchmarked against WebP instead of (obviously much worse)
JPEG?

~~~
LeoPanthera
Or indeed HEIC.

A lot of so-called "next generation" codecs are really "current-generation"
now.

~~~
roca
Benchmarks against HEVC and WebP are included in the metrics section of the
article.

------
colbyn
Interesting. Personally, I’ve been using VMAF extensively in my imager.io
project. With regards to only testing the luma channel, someone submitted a PR
for such some time ago yet it’s never been merged; I was under the impression
that the UV channel isn’t significant (in practice). Personally, I think VMAF
is great for images. For me, the biggest downside of VMAF (implementation
wise) is that it’s not currently thread-safe.

------
londons_explore
As far as I can see, there is no support in any web browsers yet, despite the
format being finalized a year ago... Whats the holdup?

~~~
niftich
AVIF support Firefox issue [1]; Chromium issue [2].

[1]
[https://bugzilla.mozilla.org/show_bug.cgi?id=1443863](https://bugzilla.mozilla.org/show_bug.cgi?id=1443863)
[2]
[https://bugs.chromium.org/p/chromium/issues/detail?id=960620](https://bugs.chromium.org/p/chromium/issues/detail?id=960620)

~~~
qwerty456127
For some weird reason browser manufacturers strongly oppose adding support for
any new media formats. They will only accept a new format once _they_ decide
that's a good political move. If I were Google/Mozilla I'd add support for
everything ffmpeg and imagemagick support.

~~~
londons_explore
As soon as they support something, they can never drop support for it because
some sites will have started using it, and some sites will never stop using
it. Look at how hard it has been to deprecate Flash.

By enabling support, they are commiting to maintain same-or-better
compatibility pretty much forever. Thats a big commitment if your code is
based of someones 'for lolz' patch to ffmpeg...

~~~
bawolff
Heck, it took until firefox 3.6 to remove xbm support, and that is an utterly
terrible image format for a web browser.

~~~
qwerty456127
> it took until firefox 3.6 to remove xbm support

Why would it be a problem to remove if only a small number of little-known
sites used it?

~~~
nitrogen
If memory serves it used to be somewhat common in Unixy land before .png was
added to browsers. There may have been long-lived documentation or academic
pages with xbm that they were reluctant to break.

~~~
qwerty456127
Then why remove it anyway? It doesn't seem hard to implement and maintain.

------
chubs
I spent some time searching for a JS shim that enables AVIF files to be used
in the browser, is anyone aware of one?

~~~
niftich
The official avif github [1] references Kagami/avif.js [2]. It's intended as a
server-side component to be installed, and uses service workers to on-the-fly
repack AVIF images as AV1 videos. The code is licensed CC0, and is easy to
read, so you can tweak it to your needs.

If the browser can't natively decode AV1 videos, it calls dav1d.js [3] to
decode, which is a webassembly port of dav1d [4].

[1]
[https://github.com/AOMediaCodec/av1-avif/wiki](https://github.com/AOMediaCodec/av1-avif/wiki)
[2] [https://github.com/Kagami/avif.js](https://github.com/Kagami/avif.js) [3]
[https://github.com/Kagami/dav1d.js](https://github.com/Kagami/dav1d.js) [4]
[https://code.videolan.org/videolan/dav1d](https://code.videolan.org/videolan/dav1d)

------
rytill
I don't see compression speed / performance / time mentioned anywhere in the
article.

~~~
aidenn0
I'm guessing that Netflix compares not at all about compression time for this;
the most used assets are probably downloaded millions of times and change
rarely.

~~~
Dylan16807
And honestly if you're not designing a camera then there's extremely little
chance you care about still image compression time.

------
herf
I'm no fan of 4:2:2 (2:1 chroma subsampling), but the comparison looks a lot
better for JPEG when you use 4:2:2 (which is the usual JPEG default). Using
4:4:4 instead drives the blocking artifacts way up, and isn't fair to JPEG.

------
jiggawatts
I'm going to put my cynical hat on for a bit and notice that the company
behind this format, Netflix, has a walled garden. They have a streaming
service that is consumed _only_ through a handful of apps that they are in
full control of. They're not doing this for the community, for web standards,
or the greater good. They're making this format simply to save some bandwidth
for themselves, in a way that won't translate to the general community. The
second picture in the linked blog article is literally a diagram of this
garden. (Titled: _" Compressed image assets destined for various client
devices..."_)

I had a similar comment about the JPEG XL format (mostly developed by Google)
and the CSS "Display P3" colour space extension (Apple), both recently
featured on YCombinator. These mega-corporations are building an ecosystem
where in 2020, _the future_ , it's impossible to send a wide-gamut, 10-bit, or
HDR still image to anyone via any of the following: web standards, chat,
email, or document exchange formats. The best you can do is send an 8-bit SDR
sRGB image and hope for the best. PC monitors, tablets, and most phones have
no _consistent_ support for colour management, 10-bit, or HDR. Televisions are
leaving the entire PC _and_ Mobile ecosystem in the dust. The closest
approximation we PC peasants have is to upload a HDR YouTube video, send a
link to it, and hope the viewer uses an newish iPhone. That's just sad, isn't
it?

The stewardship of web standards by Microsoft (#2 biggest company), Apple
(#3), and Alphabet Group (#4) have led to this. Now Netflix (#50) wants to
throw their unnecessary format into the fray, almost completely ignoring the
presence of JPEG XL and HEIC. They mention these alternatives in passing, and
then notably don't compare image quality, features, or the compression ratio
of those to their own format. You see, JXL is a _Google 's thing_, HEIC is an
_Apple thing_ , and AVIF is a _Netflix thing_. So we're going to end up with
as many image formats as there are walled gardens. I bet you too can't wait
for whatever image format Facebook comes up with specifically to reduce their
CDN bandwidth utilisation of Instagram pictures... only.

Notice also that their "idea" of making an image format readily available is a
_docker container_ , which is the most insane thing I've ever seen. Where's
the Photoshop plugin? The Lightroom plugin? The Windows 10 image codec? Oh
wait.. those are _Adobe and Microsoft things_ , so... nowhere to be seen.
Netflix Pty Ltd is not in this game to help someone _else 's_ walled garden.

No still image interchange for you peasant! Sit down, stay in the garden, and
stream that content...

~~~
ksec
First,

Jpeg XL is not a Google thing, and neither is HEIC an Apple thing. Nor the
AVIF part.

AVIF is from Open Media which is indeed a Google Thing. Netflix just happen to
take side with OM. JPEG XL is done by a Google Employees as well as other
contributors put forward to the JPEG committee. i.e It is not a Google _only_
format.

HEIC is really just HEVC Codec in an image format developed by _Nokia_ and
submitted to MPEG ( Not to be confused with MPEG-LA ).

So really JPEG XL isn't a walled garden. And support everything you described
including HDR. The problem is Jpeg XL doesn't do so well in small image size
as compared to HEIF or AVIF.

~~~
jiggawatts
> Jpeg XL is not a Google thing, and neither is HEIC an Apple
> thing...<snip>...Employees as well as other contributors put forward to the
> JPEG committee.

"Technically open" doesn't help much if each of those formats is primarily
pushed by one mega-corp, optimised for their specific use-case, and no
industry-wide effort materialises to establish widespread interoperability.

Just last week I tried to convert raw photos to HEIF images on Windows so I
could share my images in maximum quality with a bunch of iPhone users. It's
basically impossible. The _only_ tool I found was Gimp, but it doesn't
properly understand colour spaces and the output looked totally wrong. I'd
need OSX or IOS to author files in this format, making HEIC an "Apple thing"
for all practical purposes.

It's a safe bet that Google will embed JPEG XL support in Chromium, right?
Will I be able to email a .JXL file to an IOS user? Will a Windows PC user
with Outlook on their desktop be able to see the image embedded in their email
without having to save it and open it again in Chrome? Can I paste a .JXL
image into a Word document and have its various features be correctly
preserved, or will this be restricted to Google Docs viewed exclusively via
Chrome?

Look, here's the thing: All of these post-JPEG formats are nice and all, but
are worthless to me and a billion other people unless there's _proper_ effort
put into mass adoption. Real industry-wide cooperation, not just token
openness.

As I said, it is 2020, and nobody has a viable option for distributing still
images in anything other than a handful of 20-30 year old formats.

> The problem is Jpeg XL doesn't do so well in small image size as compared to
> HEIF or AVIF.

So then why isn't NetFlix working with the JPEG XL group to add the AV1 codec
as one of the encoding options? Why are they instead creating yet another
standard with no adoption likely outside of their walled garden?

Conversely, why are all the post-JPEG formats such utter failures? Take a look
at the list:
[https://jpeg.org/jpeg/index.html](https://jpeg.org/jpeg/index.html)

    
    
        JPEG         - The "popular one"
        JPEG XT      - Never heard of it
        JPEG-LS      - Never heard of it
        JPEG 2000    - Heard of it, never used it
        JPEG XR      - Never heard of it
        AIC          - Heard of it, never used it
        JPEG Systems - Never heard of it
        JPEG XS      - Never heard of it
        JPEG Pleno   - Never heard of it
        JPEG XL      - Not yet useful, will likely never use it
    

Clearly, whatever they JPEG group is doing, _it 's not working_. They got
lucky once, and in the 27 years since have failed to reproduce that success.

This is the core point of what I'm saying: The reason that GIF, PNG, and JPG
got widespread support has more to do with _luck_ rather than the
standardisation _approach_. The laundry list of failures and the sad current
state are evidence that _luck is not a strategy_ and something different must
be tried.

I put this in the same category of failure as IPv6. The IETF got _lucky_ with
IPv4 and figured that "more of the same" will work. They didn't get lucky the
second time.

I wish I had the "political clout" to bang some heads together at these
megacorps, make them sit down together and fix this.

Realistically, I foresee another decade of 8-bit, SDR, sRGB images stretched
incorrectly to whatever random gamut the $150 monitor of the recipient is
capable of displaying.

~~~
ksec
>"Technically open" doesn't help much if each of those formats is primarily
pushed by one mega-corp,

Then no Format is Open by your standard. Apple or Google refuse to support
anything would instantly make that format not Open by your standard. And if
Apple drop Jpeg, would Jpeg not be an open standard ?

What you are describing is politics, nothing to do with the standards itself.

>The only tool I found was Gimp

That is the problem with the tools. If Gimp Dev decide to take side in the
format battle and not support it properly, then is that the fault of a
standard?

>So then why isn't NetFlix working with the JPEG XL group to add the AV1 codec
as one of the encoding options?

Because it doesn't work like that.

>Clearly, whatever they JPEG group is doing, it's not working.

They are used in space other than the Web.

~~~
jiggawatts
> What you are describing is politics, nothing to do with the standards
> itself.

Standards are about convincing a large group of people to do something for the
common good, sometimes going against their most direct best interests. Nearly
a textbook definition of the word "politics".

> They are used in space other than the Web.

Yeah, the tiny, borderline non-existent space that is outside of: desktops,
phones, tablets, web, email, collaboration, digital cameras, desktop
publishing, or just about any mass-market use of computing you care to name
that involves imaging.

The _only_ usage of a JPEG standard other than the original JPEG I'm aware of
is JPEG 2000. It has some niche uses for digital movie distribution to
cinemas. Not streaming, not cable. Cinemas. Only.

According to Wikipedia, I missed a few uses:
[https://en.wikipedia.org/wiki/JPEG_2000#Applications](https://en.wikipedia.org/wiki/JPEG_2000#Applications)

I suspect a few of those uses are actually out-of-date now and have been
superseded by more modern codecs.

Flip through that _very short list_ and explain to me how the current
"politics" of imaging standards bodies has succeeded, because I just don't see
it.

------
shmerl
Why use HEIF container for AVIF, instead of something derived from Matroska
for example, like WebP does? Is HEIF free to use?

~~~
giantrobot
The HEIF and HEVC container are the same. So if they use the HEIF container
they already have libraries on devices that can handle the format, metadata,
etc. I don't believe Netflix is delivering any Matroska content so they have
no media pipeline or tooling for the format. It's the same reason they didn't
base the container format on RIFF or ASN.1.

~~~
shmerl
Since they are proposing it as a common standard, I don't think it should
matter what they already use, in comparison with something that's actually
free to use.

If HEIF is free, then fine. But MPEG stuff is not created free by design, so
it has to be stated somewhere.

~~~
giantrobot
The HEIF container is the MPEG-4 Part 12 ISO container, the same used for MP4
video and JPEG2000. This lets it fit inside existing workflows and media
stacks. AVIF just ends up and additional codec in a familiar container.

Anyone needing to pay license fees for MPEG-4 Part 12 format patents (if they
exist) is already covered by usage for video or HEIF. Anyone not needing to
pay licenses will continue to not need to pay licenses in they use AVIF.

~~~
shmerl
Not sure what that means. Sounds like some still might need to pay for it.
That's an unacceptable approach for a standard proposed by AOM which stresses
the royalty free focus (in contrast with MPEG), which was exactly my point
above.

~~~
giantrobot
I don't know of any patents on the MPEG-4 Part 12 format. They might exist but
I don't know of them. If you implement AVIF you'd have to pay whatever
licenses might exist over the format. If there's no licenses over the format,
you pay nothing. If there are licenses but you're already distributing MPEG-4
video content, JPEG2000, or HEIF your licenses you'd pay for those would cover
your use of AVIF in the MP4 container.

So to my knowledge the ISO MP4 container doesn't require license fees so
there's no real downside for using it as the container for AVIF.

