
Visualizing and Understanding JPEG Format - yasoob
https://github.com/corkami/formats/blob/master/image/jpeg.md
======
andrewgleave
Check out Unraveling the JPEG from Omar Shehata on Parametric Press[1] for an
interactive essay on the JPEG format.

Very cool to be able to manipulate the bytes directly and seeing how it
affects the image.

[1] [https://parametric.press/issue-01/unraveling-the-
jpeg/](https://parametric.press/issue-01/unraveling-the-jpeg/)

------
BlackLotus89
Hexfiend (open source hex editor for mac) [0] and synalize it [1] (closed
source hex editor for mac) both support a similar feature (hexfiend only in
the newest github release) that shows the structure of known file formats
similar to what wireshark [2] does for network packages.

Thought about writing an hex editor or adding the hex fiend functionality to
other hex editors based on the templates of hex fiend.

[0] [http://ridiculousfish.com/hexfiend/](http://ridiculousfish.com/hexfiend/)

[1] [https://www.synalysis.net/](https://www.synalysis.net/)

[2] [https://www.wireshark.org/](https://www.wireshark.org/)

~~~
tambourine_man
I love Hexfiend, so I was checking wireshark.org to see what you meant, since
I don't use it nearly as much. First, Cloudshare is evaluating my browser to
see if it's genuine access. Why? Second, _scrolling_ the site uses 97% of my
CPU. WTF is it doing? Some parallax effect on the shark, that could easily be
done with a few lines of CSS? Lost interest at the spot. What have we turned
the web into.

~~~
BlackLotus89
Can't reproduce the problem on the site, but here

> Wireshark is the world’s foremost and widely-used network protocol analyzer.
> It lets you see what’s happening on your network at a microscopic level and
> is the de facto (and often de jure) standard across many commercial and non-
> profit enterprises, government agencies, and educational institutions.

From the website. Btw wireshark is is one of the most useful tools for system
administrators (especially in combination with tcpdump and ssh) and giving up
on a project, because your browser bugs out is kind of bad form. You should at
least take a look at it. Anyway hope you change your mind and have fun ;) some
of my most interesting nights I have spent with wireshark

~~~
tambourine_man
Thanks. I know how awesome Wireshark is from the brief interactions I've had
in the past. But their website experience the terrible.

It's much more a critique of the state of the web than of the project itself.

BTW, both behaviors (insane CPU usage while scrolling and Cloudflare checking)
are reproducible in Safari, Chrome and Firefox, so it's not the browser that's
bugging out, but the website scroll highjacking that's poorly written.

------
yasoob
This repo from the author is full of gold. Shows the details and visualization
of various file structures (including multiple zip formats, image formats and
executable files)

[https://github.com/corkami/pics](https://github.com/corkami/pics)

~~~
weinzierl
Checked it out and the name _Ange Albertini_ rang a bell. I fondly remember
his _Funky File Formats_ talk from 31C3. More gold from him here:

[https://media.ccc.de/search?p=Ange+Albertini](https://media.ccc.de/search?p=Ange+Albertini)

The _Preserve Arcade Games_ talk is also good, I only realized now that both
talks were by the same person.

------
ache7
By the way, you can decode JPEG with less artifacts using these projects:

[https://github.com/ilyakurdyukov/jpeg-
quantsmooth](https://github.com/ilyakurdyukov/jpeg-quantsmooth) (good for most
mid-quality and high-quality images)

[https://github.com/victorvde/jpeg2png](https://github.com/victorvde/jpeg2png)
(use for very low-quality images, slower)

jpeg-quantsmooth gives a sharper output (use "-q6" option).

jpeg2png output is a little blurry, but it does a better job of unblocking
very low-quality images.

~~~
astrange
ffmpeg/mplayer has some more principled smoothing filters than these
(especially spp); they're made for MPEG4 but JPEG is pretty similar.

Another algorithm that should work but I haven't seen tried is edge-aware
chroma upsampling. libjpeg uses nearest neighbor upscaling for the subsampled
color planes, which is why JPEG looks so bad when there's a transition from eg
saturated red to black.

~~~
ache7
Both of these projects have chroma upsampling.

~~~
astrange
All JPEG decoders have chroma upsampling. Here jpeg2png only seems to actually
optimize it though; seems like it works but optimizing for "smoothness" could
maybe be improved.

I'm suggesting something like eedi3
([http://avisynth.nl/index.php/Eedi3](http://avisynth.nl/index.php/Eedi3)) but
guided by the luma plane.

Oh, you can also jointly optimize the YUV-RGB conversion because if the pixel
ends up out of range in RGB space it is probably a compression artifact. I
read a paper on this once but haven't seen it implemented.

~~~
ache7
"All JPEG decoders have chroma upsampling."

Well, I meant better upsampling than bilinear interpolation, which blurs
chroma.

"Oh, you can also jointly optimize the YUV-RGB conversion because if the pixel
ends up out of range in RGB space it is probably a compression artifact."

Probably not, I suspect that the mozjpeg encoder may use this, so images with
black on white text look better (artifact noise go beyond of range).

------
userbinator
If you're looking for a fun project that can be completed in a weekend or
less, one of the things I recommend is to write a (baseline) JPEG decoder. The
official spec (linked from the article near the top) is quite readable as far
as standards go, and even contains helpful flowcharts for basically the whole
process. When I did it, it took less than 1kLoC of C, including some minor
optimisations like array-based Huffman decoding (although the version in the
spec is quite compact, it is slower --- but still not as slow as the
"theoretical" bit-by-bit tree-traversal that a lot of other tutorials on
Huffman show.)

H.261 (a video format) is roughly of the same if not slightly less complexity,
and also makes a good weekend project.

~~~
taeric
Where do you get the spec for video? That really something that can be done in
a weekend?

~~~
basementcat
[https://www.itu.int/rec/T-REC-H.261-199303-I/en](https://www.itu.int/rec/T-REC-H.261-199303-I/en)

~~~
userbinator
If you can write a JPEG decoder in a weekend, H.261 should be doable. It's
much simpler in many ways --- hardcoded Huffman tables and dimensions. The
spec is much shorter too.

------
xchip
I wish I had had this when I wrote my 250 LOC jpeg visualizer!

[https://github.com/aguaviva/micro-jpeg-
visualizer](https://github.com/aguaviva/micro-jpeg-visualizer)

------
UglyToad
This is great, I was literally banging my head against a wall last night
trying to determine how the length of the ECS was determined. What a lucky
coincidence!

I'm trying to build a small C# library to support reading image types to learn
about them, png was quite understandable but jpg is a little trickier, given I
panic when I see maths.

------
gruturo
Very nice read, thanks for posting it!!!

I also learned a lot from the various JPEG articles on ImpulseAdventure (no
affiliation), e.g.:

[https://www.impulseadventure.com/photo/jpeg-
quantization.htm...](https://www.impulseadventure.com/photo/jpeg-
quantization.html)

[https://www.impulseadventure.com/photo/jpeg-
compression.html](https://www.impulseadventure.com/photo/jpeg-
compression.html)

[https://www.impulseadventure.com/photo/chroma-
subsampling.ht...](https://www.impulseadventure.com/photo/chroma-
subsampling.html)

They also have a lot of practical stuff (which chroma subsampling will
Photoshop actually use when you save at 80% quality?) in addition to the
theory.

------
ChrisMarshallNY
This is great!

The model used by JPEG is not that unusual in image processing, so this can
help us to understand many types of processes.

This person has a whole bunch of this stuff! Gold!

[https://github.com/corkami](https://github.com/corkami)

------
donatj
They could also note how 3D MPO images work, they’re just two jpegs
concatenated together with some special markers. It’s been a while but I’ve
written MPO parsers for a couple languages.

------
0-_-0
I have a question in case someone knows:

If I wanted to develop a custom JPEG decoder, what would be the easiest way to
get image data from a jpeg file into Tensorflow (or numpy, Pytorch, etc.)
tensors? I mean the raw DCT coefficients, and the coefficient table?

------
numlock86
> lossless storage:

> to make JPEG store data losslessly: use grayscale, 100% quality, then either
> width or eight to 1 pixel, or duplicate the padded data 8 times (JPEG images
> are stored in 8x8 blocks).

Interesting. I wonder if this is used internally in any applications.

~~~
ache7
Actually - JPEG is not lossless even at 100% quality, there is a little noise
at +/-1 of pixel value, some rare pixels at 2. Grayscale is not needed, you
may skip RGB->YUV conversion and encode in RGB colors, libjpeg can do that.

------
the_dripper
If I'm being honest, I'm more of a PNG fan.

~~~
astrange
PNG is a pretty poor file format, just because zlib (a byte-level text codec)
is not a good compressor for image data. Any lossless video codec works much
better.

------
tinkertamper
Where’s all the Hard G “Jerraffic” folks in this thread fighting for the
pronunciation [JAY-feg]?

------
busterarm
When you know it's going to be Corkami before you even click... :D

------
fasfsafsa
why i still see jpeg in 2020 ?

~~~
pornel
It works incredibly well for a format its age. It's simple, and widely
supported.

Internally, JPEG has the key techniques required to compress photos well, and
nothing else. Most newer formats are based on the same principles, except each
bit is upgraded to the point of diminishing returns.

HEIF and AVIF compress twice as well as JPEG, but in terms of complexity and
computational requirements are hundreds of times more complex.

~~~
busterarm
Also, in addition to what you mentioned, we have jpegs because photographers
are still shooting jpegs with their cameras.

The economics of the camera market right now will make adoption of new formats
(and the additional/more expensive hardware in the camera to support it) slow
and jpeg is a very reasonable format for photographers to shoot when they
don't want/need to shoot raw.

------
ebg13
> _JAY-peg_

I think they mean JAY-pheg.

