Hacker News new | past | comments | ask | show | jobs | submit login

> What does he mean by using native YUV formats?

Your display uses 3 bytes per pixel. 8 bits for each of the R, G, and B channels. This is known as RGB888. (Ignoring the A or alpha transparency channel for now).

YUV420 uses chroma subsampling, which means the color information is stored at a lower resolution than the brightness information. Groups of 4 pixels will have the same color, but each pixel can have a different brightness. Our eyes are more sensitive to brightness changes than color changes, so this is usually unnoticeable.

This is very advantageous for compression because YUV420 requires 6 bytes per 4 pixels, or 1.5 bytes per pixel, because groups of pixels share a single color value. That's half as many bytes as RGB888.

When you decompress a JPEG, you first get a YUV420 output. Converting from YUV420 to RGB888 doesn't add any information, but it doubles the number of bits used to represent the image because it stores the color value for every individual pixel instead of groups of pixels. This is easier to manipulate in software, but it takes twice as much memory to store and twice as much bandwidth to move around relative to YUV420.

The idea is that if your application can work with YUV420 through the render pipeline and then let a GPU shader do the final conversion to RGB888 within the GPU, you cut your memory and bandwidth requirements in half at the expense of additional code complexity.

Wikipedia is a good source of diagrams and details that explain this further: https://en.wikipedia.org/wiki/YUV#Y%E2%80%B2UV420p_(and_Y%E2...

In Commandos 2 we stored the huge map images in YCbCr 420 in memory, and converted to RGB during rendering via lookup tables rather than shaders (this was back in 1999, all CPU!). We compressed the range of each component below 8 bits, running a histogram so our bits represented values inside the range actually present in the original image. We got it down to 6 bits per pixel or something close to that.

Converting via lookup tables, one for each component, made it very cheap to perform palette-style tricks like tonemapping and smooth color shifts from day to night.

[Edit: it may have actually been CIE Lab, not YUV/YCbCr, because the a&b tended to have narrower ranges. It's been too long!]

The maps in Commandos 2 were beautiful and distinctive. I spent many hours playing through those levels.

If they were hard to play, I can assure you they were MUCH harder to create. I look back and find it hard to believe how much blood and tears the artists and level designers had to sweat. All the kudos go to them, but I admit I felt very happy and proud when I finally found the right tech to make them justice at runtime.

Do tell more! I would be curious of any details on the development of commandos2, I used to play that game a lot.

That doesn't really make sense to me, even in pipelines that deal only with YUV (very common in professional video) you always upsample to 4:4:4 in a linear colorspace and you convert back to 4:2:0 or 4:2:2 at end.

How would you do even trivial things like color blending in 4:2:0?

The implication is that you would keep the data in its original form (or 420) only until you have to perform an operation to output a new image/rendering. You could then perform the operation (such as shading operations) in a larger color space on a graphics card, and finally afterward output again, perhaps as compressed YUV or perhaps as RGB over the wire in 8-, 10- or 12-bit (as you get to fancier HDR10+ and/or Dolby Vision). (Note in this post I'm using YUV instead of the more technically correct YCbCr...)

That said, this (storing JPEGs in YUV420) is just an optimization and the more images and displays go HDR, the less frequently we'll see YUV JPEGs, though we could see dithered versions, maybe, in 444. That's basically the same thing, but once you discard 420 and 422 you might as well use standard 8-bit RGB and skip the complexity of YCbCr altogether. If you're curious about HDR as I was, though 10-bit is "required", you can dither HDR to 8-bit and not notice the difference unless doing actual colour grading (where you need the extra detail, of course). For obvious reasons, I've never heard of anyone dithering HDR to YUV 420 though, and most computer screens look pretty terrible in when output to a TV as YUV422 or YUV420.

Real-time games are bandwidth limited. 100% increase is a real cost. Maybe worth some awkwardness in the engine.

Wow, you explained that incredibly well, thanks! I knew nothing about what Carmack was talking about here, but your explanation makes it pretty clear.

In what way is RGB easier to manipulate? I get that it's the format we have to send to the monitor. Also, current tools may be geared more towards it. But on the face of it, I don't see why it is simpler. Do you have any good examples?

I can think of several image processing tasks which are more straightforward in a luma/chroma format. Maybe it's because I'm more used to working with the data in that form?

>In what way is RGB easier to manipulate?

Probably because it's the native input format for every display technology in existence? If you're going to twiddle an image, you might want to do it in the format that it's displayed in.

Mostly the issue is the layout, not the color space. YUV420 is awkward to handle because it's "420", not because it's YUV. There wouldn't be too many issues if everything ran on the 1-byte-per-pixel-component YUV444, besides converting to/from RGB for certain processing steps.

It's quite common to upsample to 444 for processing, but using 420 for storage.

This was such a good explanation, thank you for taking the time.

I think a twitter account called @JohnCarmackELI5 that just explains all of Johns tweets like I have no idea what he's talking about would be invaluable. The man is obviously bursting with good/interesting stuff to say, but I grok it like 5% of the time.

Thanks, this is a very clear explanation.

That was an excellent explanation, thank you.

Thank you for the clear explanation. I know little about graphics, but this was easy to follow.

Applications are open for YC Winter 2022

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact