This technology could be used to cut down on files sizes, to raise video quality and to up resolution. A few thoughts:
1. 1080p streaming video from Apple, Netflix and others looks pretty good, but it suffers from more compression artifacts when compared to Blu-ray. It simply doesn't look as good. With a more efficient compression technology, streaming video could have similar file sizes to today's H.264 video but look more like Blu-ray.
2. I already own a laptop that does above 1080p video. When Apple release a Retina iMac or Thunderbolt display it will be around 5k. Other manufactures will be there too. 4k or so TVs and projectors are on their way. Streaming technology makes a lot more sense than a new physical format for higher resolution video. In order to realistically deliver 4k video (or 5k video like The Hobbit is being shot in) over IP, we would need better compression than H.264
3. Files sizes could be cut down, allowing people to consume more video without going over their caps. This would also allow mobile devices to store more high quality video content for on the go.
The best case scenario would be that we get a combo of all three. 4k-5k video content is still a bit way for home use, but when it does come. H.265 sounds like the way to go.
In the next few years, this technology could be used to cut down on file sizes some, while also upping the quality of video. This is what we saw with Apple's 1080p video which uses high profile H.264 video, whereas Apple's 720p video is main profile H.264. Yes, the 1080p videos are bigger, but not by much.
Nitpick, but Thunderbolt supports a maximum of 20Gbits/sec which can't drive 5k. It maxes out at 10 megapixels which is about 4k. 5k is nearly 14 megapixels.
It'll be interesting to see what Apple does then because 2X the current Thunderbolt Display (how Apple has done hiDPI)would put it above 5k.
Maybe it will take a future version of Thunderbolt using optical cables to do a Retina Thunderbolt Display. Or maybe Apple will simply make it around 4k.
Apple do the 2x stuff on iOS so that the large set of handcrafted, bitmapped 1x apps can map precisely to pixels when you upscale them. OSX doesn't have the same issue, since there's no existing default screen size that is designed for.
Since Thunderbolt is also some sort of PCIe port, couldn't you have the GPU be internal to the Cinema Display, and transmit only the GPU code over Thunderbolt? Or are significant parts of the screen drawn by the CPU directly these days?
I suppose that would be possible, if the built in GPU has local video RAM and such. You wouldn't nearly be consuming the bandwidth compared to sending raw video to the display.
I do think it would be a little too complicated though, it's probably more likely there will be a faster Thunderbolt port before that happens ;-)
Doesn't that assume no compression in the transfer? Surely, if streaming compression is getting better, there's some lossless compression that can be used for display signals?
Generally these things would have to be done in hardware, bumping up the price by a non-negligible amount. Additionally, it would increase the input lag on the monitor, something manufacturers are trying to avoid.
The new Macbook Pro Retina has two Thunderbolt ports. In theory, they could build a display using both, each driving half of the screen. Although it's unlikely to happen.
Would that not be pretty similar to dual-link DVI? Though I believe that was part of the DVI specification whereas I've never heard the current Thunderbolt standards to have provision for such. Regardless, I wouldn't be shocked for Apple to do it at least initially in their own non-standard way.
Maybe it's just me, but I don't need more than 1080p resolution for watching a video. I don't recall the exact numbers, but I think you need to be within 5 feet of a typical 42" screen to notice an improvement of 4k over 1080p. That's bit close for my taste! Now if you have some kind of enormous video setup that would benefit, then more power to you.
People would either need bigger TVs and/or to sit closer. It's the same with 1080p vs. 720p.
Ironically, I might be better able to make use of 4k video on a computer since I sit so close. I find video to be a bit soft on the iPad 3 and Retina Macbook Pro because both have to up-convert 1080p video. I expect to have a 4k or so 27-inch monitor in a few years, and would love to be able to watch native video on it.
I would need to both sit much closer and need a bigger TV to make use of 4k video. I still have a 720p plasma because where my couch is, it's almost impossible to tell the difference between 720p and 1080p.
Some of the detail may have been lost in the compression and some in the upconversion so better source compression and better upconversion may help a lot although more resolution is always nicer.
Yeah, I understand why movie theaters are interested in 5k, but I don't think most people will be able to see the difference on home televisions.
Not that that will stop them from making them, but it'll be interesting to see what happens after 4k/5k when the limit of how much resolution the human eye can perceive has been surpassed.
Marginal improvements in apparent resolution aren't the only thing that a more efficient codecs can deliver. The same capacity can also be used to increase the bitrate of the encoded files dramatically, and / or deliver streams at much higher frame rates. In terms of improving picture quality, jacking up these aspects while keeping resolution at less than the supported maximum can be far more effective places to spend your bit budget.
I have recently acquired weak glasses (0.25). I wear them when I'm watching good HD sources and take them off if there were significant artefacts or low res source as they are better slightly blurred and there is the comfort benefit.
I do care about video quality although I know that I'm in the minority (many people say that picture quality is the most important thing in a TV but very few of them can actually identify it). Given that my very weak prescription makes such a difference (720P and 1080P content is probably the same without my glasses) it wouldn't surprise me if many people really couldn't see it.
In the majority of situations, most people cannot tell the difference between 720p and 1080p. Of course, people who are interested in home cinema type setups ("videophiles"?) will be able to tell immediately.
I think that a major change you will see in the future is the size of the screens. In order to take advantage of a 4k stream a monitor needs to occupy a much larger fraction of your vision than 1080p. So 3 metre screens or projections in people's homes will become more common if resolutions are going to meaningfully increase.
The missing words are "well encoded 720p and 1080p". The difference between bandwidth starved cable and over-the-air HD is amazing. Having a new standard that can encode a better picture over cable or satellite is more than welcomed.
I'm not at all familiar with video encoding/decoding processes or what they entail. Can you explain how hardware support for it would drastically change power requirements?
Sure, specialised hardware tends to be more efficient than general purpose hardware but it's not so much "special-purpose hardware is magic" as it is "the general purpose hardware doesn't take advantage of certain properties". As far as I understand the GPU's advantage over CPU (again, graphics is not my area of expertise) it's largely that graphics processes are massively parallel and the GPU is specifically set up to handle massively parallel things. Like I said, I know nearly nothing about video encoding/decoding so I was hoping someone who did could explain things for me. My only guess for how specialised hardware could dramatically save power for video decoding is that it might be comprised of massively parallelisable simple operations.
Hmmm.... I'm not very versed in video encoding technology but from what I gather x264 has been adding features outside of the 'official specs' for pretty much all it's existance, and looking at the list I directly recognize things like higher bit depth and cabac from x264 settings.
So I'm wondering is there some new sercet sause in h265 which makes it really much better than say x264 or is it just a new standard created around extra features which encoders like x264 already has implemented?
H265 has some quite innovative features (over H264), for instance instead of dividing the picture into rows of macroblocks, it's divided into quadtrees, which allow the compression algorithms to make more use of spatial similarity.
As far as I can tell they just amped up similar features from h.264. Better motion estimation, more block types in B-frames, etc. Just "more" of the same. edit: from that link I posted larger block sizes, larger transform sizes, fancier interpolation filters, improved intra prediction schemes, improved motion vector prediction, increased internal bit depth, new entropy coding schemes, and so forth.
x264 does not break spec. It looks familiar because H.265 is a very incremental improvement over H.264. I suspect the biggest single difference is the larger transform sizes.
That's from over 2 years ago. A version 10x faster (and with half the code stripped out) was available last year and I assume someone has made a sane version since.
Anyways he explains what happened: instead of choosing sane options, that encoder would iterate through every possible encoding and test each of them for PSNR, for a competition. It's not the kind of encoder you would ever use in practice.
For reference, there are 129,600 frames in a 1.5 hour long 24 fps movie.
At 0.75 hours per frame, that's 97,200 machine hours, or just under three years even if you have a cluster of 100 machines.
This might be GPU encoding's chance to shine. Right now it's generally considered to not be worth the hassle over a fast i7 machine.
And of course the algorithms will be optimized, maybe by orders of magnitude. But that's still a lot of computation power required, no matter how you slice it.
It's worse than that, as the algorithms are not simply parallelisable (frames relate to each other). Of course you could chop the film into 100 pieces, and compress each individually, but you would lose some compression rate which would (partly) defeat the whole purpose.
Not really - the encoder tends to reset quite often, e.g. it is recommended to insert an "I" frame every 5 seconds or less (stands for "intra coded", but could be thought of as "independent") - so that you have to decode at most 5 unused seconds when doing a random seek.
You wrote a whole lot here based on years-old numbers for a prototype encoder whose implementation wasn't even relevant when the numbers were posted, as acknowledged in the posting. Why did you do that?
I participate in the GSoC and my project is an HEVC decoder for libav: https://github.com/smarter/libav/tree/hevc (the decoder is contained in the libavcodec/hevc* files). It currently only decodes I-frames and doesn't include the in-loop filters.
Assuming this is true across the board and not only for some edge cases, will this actually mean that video files get smaller? It seems to me that this is a case of Jevons paradox[1] where increased efficiency leads to higher consumption. Example: x264 led to 720p encoded videos with higher file size rather smaller files with lower resolution video content.
A lot of the 10-bit H.264 videos out there are smaller than they would be if the same people had released them with 8-bit encoding. At some point, the quality difference between the encoded version and the source video becomes negligible, so it's natural to expect that better compression would result in smaller file sizes for any given raw source.
The time when this doesn't apply is when you're currently losing a noticeable about of video quality to keep file sizes down. Say that you're a movie streaming service, and you want to stream a fairly new movie. You have it at very high resolution, but if you wanted to stream it at Blu-Ray quality, that would really eat up your bandwidth, so you decrease the file size to something you're comfortable with, and get as much quality as you can within those limits. Better compression, in that case, would mean either better quality at the same file size, or bandwidth savings at the same quality, or some combination of the two.
So: I would expect this to make pirated TV rips smaller, pirated Blu-Ray rips larger or smaller (depending on what the encoding is optimized for), and streaming video either higher quality or smaller, depending on your connection speed and how much bandwidth they're willing to pay for.
Users will vote with their feet. It was remarkable how much the filesizes of videos from certain sources fell in the aftermath of the Thai floods and increased consumer hard drive prices.
Of course, there's a class of users who will just always find the largest video file possible, reasoning that it much be higher quality.
I'd say that's a good possibility, for now. Consider when Apple rolled out 1080p downloads - the filesizes were only slightly larger than for the previous 720p versions, representing quite an efficiency boost. Adopting H.265 would seem to be able to chop the sizes down.
However:
- tablets and phones are unlikely to be able to take advantage of H.265 until the requisite GPU support arrives. This may hold up widespread adoption for a year or two.
- for how long will the video arena remain at 1080p? I hear Sky (UK satellite broadcasting arm of the Murdoch empire) is trying to nudge toward 4K broadcasting.
It's a somewhat backwards way of thinking about it. They chose the filesize and then set the dials for the encoding. They could have chosen the 720p to be smaller larger or the same size compared to 1080p.
I'm curious what kind of hardware will be required for playback-- my homebrew'd Wii already has trouble playing back highly compressed h.264 videos. I wonder how much more power will be required to decode high-motion scenes in real time with h.265?
I don't think the files will get larger before the displays get larger. I haven't seen many QHD (quad Full hd) or UHD (16x Full HD) displays. I think you can buy them only in Japan now. But even the new mac book display has by far not enough pixels for that, so i doubt this will ever be adopted on laptop screens, it just seems unreasonable, nobody will be able to tell the difference.
Sweet! Half the bandwidth, double the copyright infringement.
I still remember the days when downloading movies was only practical because someone had compressed them down to two 150-megabyte videos. When I got my first 'high quality' 580-megabyte copy of The Matrix, I was thrilled.
Encoding used to be an art form. Now people just use whatever codec they want with default settings to get that 50GB Blu-ray movie down to a couple gigabytes and call it a day.
I'm glad it's so easy today. "Back in the day" you had to rely on all sorts of oddball codecs to get something to play. It's the reason why VLC was (and still is) so loved - it could play anything.
Using a standard means that I can take a video I encoded with an off-the-shelf tool and play it on any device.
I remember those days; vivo was state of the art, then either realplayer g2 or MS-MPEG4 .asf files. Eventually someone hacked the MS-MPEG4 codec to allow using in .avi files (it was artificially restricted to be only in .asf files). Then someone (vulture IIRC) rewrote the color conversion scheme to use MMX while increasing its accuracy at the same time; remember, this is with a binary, not source.
And of course you had to get the encoding settings right before encoding the whole thing, since the moment a VCD was public there were several groups all working to get the first decent quality small video file out there.
Yeah, I'm mostly curious how this codec stacks up against WebM/VP8 or whatever other free codecs are out there. It'd be nice to see something take off that's less encumbered by licensing issues.
H.264 is already superior to VP8. I see no reason to suspect that this situation will change.
EDIT: I mean to say, H.264 implementations (particularly x264) are way, way, way out in front of VP8, so even as the H.264 standard is better, the actual facts on the ground are even more so. The only reason to use VP8 (and let me be clear: I think this is a perfectly legitimate reason) is political.
Every generational codec announcement from MPEG manages to attain "50% improvement", so it really remains to be seen. It certainly is a more advanced codec than H.264, which itself is already rather better than VP8 in its present form.
Regarding the other thing, MPEG is running two tracks for a royalty-free spec based on existing patent-free tech and on grants from H.264 patent holders (respectively) which they say they will decide on sometime this year. An option like that from MPEG might not take off but can't hurt.
I think that you probably get 50% at the generation introduction and another 50% over 10 years as the encoders are improved (and more hardware is thrown at the problem).
Double track on licensing makes some sense for a profile that can be served or provided free but I expect the efficiency benefits would mean that for commercial uses you would pay. That would mean that most hardware will support both so really I think most decodes will support both.
At this point, you are talking hardware acceleration is a must because of mobile (and it seems encoding on this one). Chip makers are fairly experienced with licensing, and a bit scarred of the unknown. MPEG-LA is a known.
"Encoding used to be an art form." It's gone downhill but there are still people keeping a high standard. YIFY is known for releasing high quality 720p h.264 Blu Ray movie rips that fit on one CD (700MB), and 1080p rips that are 1GB. Their stuff looks good, better than cable HDTV or Netflix to my eye.
They surely plan to lock the industry into their new closed codec for another long time, since patents on H.264 will eventually expire. Will anyone come out with improved open codecs to counter that for the sake of open Web?
on that note, does anyone know what patents for h.264 are still valid, and for how much longer? I've been looking for that info for quite a while now, without success.
So 15 more years, basically. Man, that's depressing. What's fast becoming the standard for HTML5 video has 15 more years of not being truly free. I wonder what mozilla will do about that.
The problem is, with these perpetual upgrades (which undoubtedly will be patented from the the present time) they want to stretch it well beyond 15 years. Luckily H.264 is not a standard in any way, since W3C removed any specific codecs from the video/audio tag spec.
Possibly new codecs won't be needed. After all storage is increasing exponentially, and even bandwidth is going up slowly. For most users it doesn't matter if a movie fits in 600 MB or 300 MB.
People will get along with H.264 just fine. If the industry (ISPs, mobile operators, Youtube, Netflix, ...) want us to use H.265 (they will gain the most from it), they will make it free.
They don't want you to get along with H.264 just fine when H.264 patents will expire. That's why they are pushing for renewed codec. And the only way such thing can become free, is if someone would buy all patents on it (like Google did for VP8 with On2), and release the codec as free. But unlike the story with On2, with so many parties involved in MPEG-LA it's simply impossible. It's their perpetual cash cow, and they want to keep it that way, while it goes against the interests of the open Web.
You are confusing MPEG the ISO working group with MPEG-LA the private company of lawyers. ISO has nothing to gain from patent trolling, they are financed by the purchase of their standards. MPEG is doing the exact same thing they have since 1988. That's saying nothing of the ITU, which is equally involved in this spec.
I was referring to the MPEG-LA (the patent related entity) not to the ISO working group. The new spec is coming from the technical participants, whose patents are managed by MPEG-LA. The patents of the resulting codec will be obviously managed by MPEG-LA as before, and they don't want the situation when H.264 patents will expire and the industry would get away from the lock in into their closed codec. They want the closed part to persist.
The intentions of improving technical side of the codec are also there, but it comes along with expanding the time of the patent grip on the whole thing.
ISO itself has nothing to gain from the patent setup for the MPEG specs, but their members sure do. If you haven't read http://lists.xiph.org/pipermail/theora/2010-April/003769.htm... before, you may want to. It's an interesting perspective on the MPEG standards process.
I'm pretty libertarian, but for things like standards and formats, I really think the government should be stepping in and taking control. Standards, formats, and basic internet access are the new "roads" of the modern world. Commerce can flourish when we aren't fighting over them.
Even for something R&D heavy like video codecs. How much money are we dumping into the NSA right now? Use some of that.
There doesn't need to be control taken; instead, the government could just buy the patents to the best codecs and open them up. Software infrastructure, if you will.
It's sad and ironic that Google is doing this more effectively than We The People, it least in cases where their interests are aligned with the commons.
Which government institution should decide on the video codec for everyone to use? How would they know when it's time to switch? I cannot see that working any better than MPEG and MPEG-LA.
That institution probably does not exist. The Federal and State governments are still trying to figure out what an Internet is. This isn't even on their radar.
For codecs, the government would not even need to regulate. As long as the offerings are good, the industry would be compelled to adopt them because of the lack of licensing costs and for the long term stability. Continuous R&D would still be necessary.
Well, for cryptographic hashes the relevant institution in the US is NIST and they switch at a point when there start to be worries about the previous hash being subject to successful attacks sometime in the future.
There's no reason in principle that the same approach, again with NIST as the relevant institurion, could not be used for video codecs.
It would be better than the MPEG-LA because the patent situation could be made much simpler (e.g. automatic patent licenses would be granted to all implementors of the standard).
Do you want Web "standards and formats" to last as long as NTSC did? Imagine using IE6 for the next 50 years. That's what government involvement will get us.
Which is why the US and EU governments are fighting against Google, Samsung and HTC who are abusing FRAND obligations and undermining the concept of standards.
The benefit will probably go first to someone like Apple or Google who can both supply streaming content and control the software (and ideally hardware) on devices (I imagine for low-powered devices you'll want hardware decoding, so this will prevent Apple from, say, adding support to existing AppleTVs).
I guess we can all complain when the iPhone 5 doesn't support it.
Google is heavily pushing VP8, which is supposedly royalty free. I'd be incredibly impressed if Apple could make a hardware decoder/encoder for HEVC for their next iPhone (or whichever one comes after the standard is finalised), but we won't know until then :)
http://www.webmproject.org/tools/vp8-sdk/
It will be interesting to see what happens with VP8/WebM. Really it looks like Google tried to stymie Apple (which committed itself to H264) first by trying to back Adobe/Flash and then VP8 (and announcing that H264 support would be dropped from Chrome, which AFAIK it hasn't been on any platform). Thus far I don't see VP8 achieving much and Google may just end up sticking with MPEG standards.
IIRC the PowerVR GPU inside all the iOS devices already support hardware VP8 encoding/decoding.
Apple just has no inclination to support it. And why would they ? VP8 is significantly worse than H.264 in every important way (quality, compatibility, popularity). Not to mention that Apple is on a very anti-Google path right now.
So it's doing it with half the minimum block size and looking far forward (and backward) in the stream.
The cpu requirements must be intense. If they cannot do it with hardware accelerated video drivers for current hardware, it sounds like it will tie up multiple cores?
Maybe they like the idea of making everyone rebuy hardware.
This claim is pretty accurate. Since the standard isn't finalised yet it will be a while before the hardware is developed to make it widely used in mobile devices, as it's improbable that a software encoder could be made that uses little power.
Well at least this time it will be much faster then H.264, Hardware decoder are already in the work with many things could be reused fro H.264 HP Decoding. So unless there are any major changes HEVC decoder will be coming much quicker. Some Video Decoder IP has already begin to list HEVC decode as a feature.
Half the bandwidth with no quality loss is a pretty bold claim. Bold enough to be unbelievable - it's not like the existing codecs are doing a horrible job. Data rates are easily measurable, so I'd guess that "no quality loss" is an exaggeration. Anybody have any data on this?
Hmm, still no overlapping blocks, apparently. Do other people find block artifacts less distracting than I do, and that's why nobody's trying to fix it for image and video compression?
Well, no, it's not - at least not in H.264. I have not read details yet on how it works in H.265, but in general deblocking is a completely different approach to solving block aliasing.
Deblocking works (very roughly) like this: look at the edges of the blocks and see if there's a sharp edge there. Now check and see how strong of an edge there is there in the original input. Depending on these two relative edge strengths, blur the block edge. That is, if there's a strong edge in the output, but not in the input, blur a lot. If the input does have a strong edge, blur less. H.264 also uses some other heuristics to decide how strong the edge filter should be, and happens to do the filtering on the encoder side as well as the decoder side, which allows for better interframe compression.
So while this, in a vague mathematical sense, does provide overlapping information between blocks in a way that can be analogized to overlapping blocks, that information is far cruder than true time-domain aliasing cancellation
But to answer my own previous question, the reason they don't use overlapping blocks appears to be that that the concept is very difficult to reconcile with motion compensation.
I think where it ends up being an especially big problem is with low-luma regions that have subtle gradients. A particularly common example is a scene showing the sky at night. I'm not sure what the crux of the issue is, but I think it may be some combination of the following hypotheses:
1. Psychovisual models are overly pessimistic about human perception of dark scenes and bit rate is reduced in these scenes too aggressively.
2. Large areas of subtle gradation cause the motion predictor to use big 16x16 blocks. This would be bad on two counts: i) the deblocker tones down to its lowest level on motion compensation boundaries, and ii) the deblocker can affect a maximum radius of 3 pixels from the edge, which would do little more than make the blocks look fuzzy.
3. Greater quantization in dark areas means that faint, transient noise like film grain is smoothed over, but the edge detector still sees it in the source frame, and that causes the deblocker to turn off (or at least weaken) on those blocks. If the noise were there, it would perceptually mask the boundary, but because it has been lost to other compression techniques, the block boundaries stand out. This particular hypothesis suggests that the problem could be reduced without modifying the spec by adding a simple luma-sensitive denoise preprocessing step.
Sort of, but you vary the weighting to fade the blocks into each other.
Or technically speaking, what you do is you multiply the blocks by a window function before you do the Fourier related transform. If you choose the windowing function carefully, you can even set it up so that all you have to do is add the overlapping areas together.
This is especially easy to understand in one dimension, which is more or less how MP3, Vorbis and AAC do it. Block boundary effects are so noticeable with audio that unless they are corrected very robustly, the quality is unacceptably choppy.
The technique generalizes basically without alteration to two dimensional data, but I've never seen an image or video algorithm that used it. JPEG just ignores the blocking issue entirely, and video algorithms seem to rely exclusively on deblocking filters. As I said, I think this has to do with the fact that TDAC doesn't trivially generalize to motion compensation.
It's likely MPEG-LA will handle the licensing and I'd expect a similar cost structure. Newer codecs have tended to be cheaper, although at this point I suspect pennies per unit are pretty irrelevant.
1. 1080p streaming video from Apple, Netflix and others looks pretty good, but it suffers from more compression artifacts when compared to Blu-ray. It simply doesn't look as good. With a more efficient compression technology, streaming video could have similar file sizes to today's H.264 video but look more like Blu-ray.
2. I already own a laptop that does above 1080p video. When Apple release a Retina iMac or Thunderbolt display it will be around 5k. Other manufactures will be there too. 4k or so TVs and projectors are on their way. Streaming technology makes a lot more sense than a new physical format for higher resolution video. In order to realistically deliver 4k video (or 5k video like The Hobbit is being shot in) over IP, we would need better compression than H.264
3. Files sizes could be cut down, allowing people to consume more video without going over their caps. This would also allow mobile devices to store more high quality video content for on the go.
The best case scenario would be that we get a combo of all three. 4k-5k video content is still a bit way for home use, but when it does come. H.265 sounds like the way to go.
In the next few years, this technology could be used to cut down on file sizes some, while also upping the quality of video. This is what we saw with Apple's 1080p video which uses high profile H.264 video, whereas Apple's 720p video is main profile H.264. Yes, the 1080p videos are bigger, but not by much.