

Bug or Feature? Unintentionally Intentional Behaviors - espadrine
https://people.xiph.org/~xiphmont/demo/daala/random.shtml

======
jwise0
This story I think very neatly points out the difference between verification
and validation -- and why it's important to have both of them in a testplan
for anything new.

In short, verification asks the question: "does it do what we designed it to
do?" For instance, if you have a reference encoder of some kind that is a
verbatim translation of a specification, and a high-performance encoder that
is designed to squeeze every last drop out of the system, you can verify the
high-performance version by comparing the output to the reference; if they're
the same, then you know that the high-performance version, at least on some
level, works as designed.

If verification compares the design against the implementation, then
validation asks the question: "is the design what it needs to be?". The story
that xiphmont told here is a validation issue: the implementation is perfectly
correct, but it's the design that isn't what he'd hoped for! Although
verification often takes more of the time than validation, in my experience,
validation often seems to take quite a lot more creativity than verification.

Thanks to Monty for writing this up, and providing such an illustrative
example of the difference! It's a useful distinction to have, especially when
looking at errors that escaped testing and made it to production.

~~~
QuantumRoar
I've had a lot of these problems where everything is correct but nothing works
as intended. Turns out, numerical mathematics excels at making programming
more about validation than verification.

Breaking the continuum into pieces breaks your symmetries. For example on a
finite rectangular grid, a sphere is not rotationally invariant. Even if your
maths is valid in the continuum, for the implementation, you still need to
prove that it is valid (i.e. fulfills all desired properties) on a discrete
grid.

I have zero experience with video compression, however, the author states that
the issues get less severe when the video is rotated by 90˚. I guess the
compression algorithm is designed to work irrespective of orientation.
Therefore, my humble guess is that they failed to implement the desired
symmetry of the continuum in the compression algorithm that acts upon discrete
spatial coordinates. Unfortunately, there's no way to recover all symmetries
from the continuum on a discrete grid. You always get some kind of artifacts
and you can only work to reduce them for the case you are considering.

------
metajack
For anyone that wants to get started hacking on Daala, we keep a list of easy,
good first bugs here on GitHub.

[https://github.com/xiph/daala/issues?q=is%3Aopen+is%3Aissue+...](https://github.com/xiph/daala/issues?q=is%3Aopen+is%3Aissue+label%3Aeasy)

It's a really fun project to hack on!

------
Pinatubo
I assume this article is written for people who already know about Daala. I
had to search on the name to learn it's a video compression technology. After
that the article made more sense, but a 2-3 sentence introduction at the start
of the article would help people like me.

~~~
xiphmont
Point taken, short intro added.

------
WhitneyLand
This article needs to be read by any executive trying to push aggressive ship
dates.

It's hard to explain why development is not always easy but I think this
provides a pretty sobering example.

------
mlinksva
I see these two links submitted yesterday (the 2nd very briefly on front page)
but no discussion:

[https://www.ietf.org/mail-archive/web/video-
codec/current/ms...](https://www.ietf.org/mail-archive/web/video-
codec/current/msg00235.html)
[http://www.ietf.org/proceedings/92/slides/slides-92-netvc-0....](http://www.ietf.org/proceedings/92/slides/slides-92-netvc-0.pdf)

Realistic that there could be an industry consensus next generation video
codec (like Opus for audio) with Daala as basis?

~~~
shmerl
I hope it will work out. It worked for Opus for WebRTC as a mandatory codec,
though I'm not sure if Opus was proposed as mandatory for the audio tag (I
think it should).

Xiph used preemptive approach and refuted all accusatoins and patent attacks
on Opus as soon as they surfaced, which helped its acceptance in IETF. They
plan the same tactic for Daala which hopefully would kill all the FUD. That's
in contrast with VPx where Google waited for lawsuits instead of defending it
right away and refuting any false accusations.

However IETF acceptance doesn't mean that the likes of Apple will actually
start using it. But making it mandatory for the video tag would be a big
victory, since those who refuse to support it will be under pressure to follow
the standard.

~~~
derf_
_> I'm not sure if Opus was proposed as mandatory for the audio tag (I think
it should)._

[https://www.w3.org/Bugs/Public/show_bug.cgi?id=18932](https://www.w3.org/Bugs/Public/show_bug.cgi?id=18932)

It was CLOSED WONTFIX.

~~~
shmerl
_> Status: Rejected Change Description: No spec change. Rationale: See Ian and
Adrian's rationale in comment 2 and comment 3._

 _> #2: Working groups can't force a browser to implement something. Are all
the browser vendors willing to implement it? That's what matters._

 _> #3 Microsoft doesn't believe HTML5 needs to specify a mandatory to
implement codec. The market is quite capable of ensuring that popular formats
are supported on the web. Opus did not exist when ISSUE-7 was closed. Who
knows what formats might be more popular than this by the time HTML5 gets to
Recommendation, or after that. It's also not clear that there will be
widespread adoption of Opus for the purposes that <audio> is currently used.

This does not mean Microsoft is for or against supporting Opus with <audio>,
just that the spec doesn't need to say anything on this._

Rationale is invalid, since in other instances standard is enforcing something
(same WebRTC). And of course MS was brought as the main antagonist. Some
apparently just hate when the Web has interoperability.

~~~
derf_
Well, there are other ways to tell them that this interoperability is
important: [https://wpdev.uservoice.com/forums/257854-internet-
explorer-...](https://wpdev.uservoice.com/forums/257854-internet-explorer-
platform/suggestions/6513488-ogg-vorbis-and-opus-audio-formats-support-
firefox) (no registration required to vote).

~~~
shmerl
Thanks for the link!

------
derf_
This is the latest in a series of posts on Daala:
[https://people.xiph.org/~xiphmont/demo/](https://people.xiph.org/~xiphmont/demo/)

------
danbmil99
Years ago we discussed the idea of penalizing errors more if they stick
around. So a difference of +2 might be acceptable for a frame or two, but a
_consistent_ +2 over several frames should be severely penalized due to the
fact that the human visual system is much more likely to notice an artifact
that lasts 300 ms versus one that lasts 30. Whereas 10 frames alternating +2
with -2 would be acceptable.

Perhaps that sort of logic is rolled into the rate distortion metrics of
Daala? But if so, I suspect this bug would not be.

~~~
derf_
Well, one has to be careful here. In a static portion of a scene, a consistent
error likely to look much better than an error that flips between being too
large and too small in alternate frames, since the latter will show up as
noise or flicker that was not in the original. That can be more objectionable
than simply being slightly wrong. Notably none of the metrics we use for
evaluating Daala have any means of measuring this.

The better approach, I think, is some form of "temporal RDO", where you spend
more bits on places that will be good _predictors_ of future frames. We
discussed the idea in a Theora update in 2010 [0]. x264 implemented it even
before then (under the name "mbtree" [1]). It is not yet implemented in Daala,
as we do not have two-pass rate control or lookahead yet. It's possible to do
one-pass, no-lookahead versions by estimating future prediction efficiency
based on the past, but you want one with full lookahead to compare against to
make sure those estimates are doing the right thing.

I am not sure temporal RDO would save us here, because the issues described
only occur in areas of the frame that are being updated. It still might have,
in this case---these things are relative---but we could construct cases where
it would fail to do so.

[0]
[https://people.xiph.org/~xiphmont/demo/theora/demo9.html](https://people.xiph.org/~xiphmont/demo/theora/demo9.html)

[1]
[http://www.x264.nl/developers/Dark_Shikari/MBtree%20paper.pd...](http://www.x264.nl/developers/Dark_Shikari/MBtree%20paper.pdf)

------
shmerl
I pity Daala got delayed to 2016, but on the other hand it should be
noticeably better than competition in order to succeed, so it's good they are
taking their time.

------
cmrx64
I don't know what it is, but the xiph.org blog posts always seem so well
written and interesting to me. Probably that I perceive the subject matter as
magic :)

