Had the pleasure of working with Jason Garrett-Glaser (better known as DarkShikari) circa 2011 on a patch for x264 to better utilize more cores (slice-threaded lookahead). At the time he was working with Sony to optimize their PlayStation game streaming service, and this had the potential to lower encoding latency (critical to game streaming) even on same-core systems by spreading the frame over more cores to do more parallel work, and doing more intelligent scheduling and convergence after that tile-based work was reconverged into a frame. At the time we were using servers with 48 physical cores to do transcoding, a bit of a rarity in a single NUMA space back then and there were no instances to test this on so I flamegraph traced a few encodes at very high thread parallel counts and send back the instrumented profiles (from a custom compiled version) showing the core work being done and he used that to optimize the motion estimation code which worked on the first shot to perfectly level out cpu usage and increase encoding speed across ~24 cores (was previously topping out around 8 and running one core hot iirc). I got a look into how exotic and complex this project was, particularly in terms how low level and advanced things like motion estimation, quantizing and perceptual compression were... it was and still is sort of magic how many smart techniques are baked into something that we barely pay attention to. They were, even at that time, working on x265.

His blog, Diary of an x264 developer, was a great read too for anyone wanting to get a vantage point on how crazy that codebase was.

[1]https://github.com/mstorsjo/x264/blob/master/doc/threads.txt

(edit: was Gaikai that was acquired by Sony in 2012 and is the basis for PlayStation Now)

One uncommonly used feature of H.264 is support for lossless video. It's a bit hard to use due to tooling (not easy to discover and select the feature), and is sometimes incompatible with web browsers, but you can get actual lossless video using a mainstream video codec. You can verify that it really is lossless, and doesn't have any conversion step where it's quantized to 8-bit YUV.

"x264vfw" is one tool which supports this format, even inside of an AVI file. So older software that uses Video For Windows can create those files.

Ten years later, 2026 magic: AV1

https://engineering.fb.com/2026/06/22/video-engineering/adop...

And soon AV2

https://av2.aomedia.org/

Previous discussions:

4.nov.2016 https://news.ycombinator.com/item?id=12871403 219 comments

17.mar.2022 https://news.ycombinator.com/item?id=30710574 219 comments

24.may.2019 https://news.ycombinator.com/item?id=19997813 180 comments

14.jun.2024 https://news.ycombinator.com/item?id=40681306 205 comments

I really like the idea that compression is intelligence, there are so many tricks an interesting ideas related to it. A bunch of them are described in the article, but you can imagine a long tail of many more inscrutable tricks chained together to get really incredible shrinkage in data.

Hutter prize (I think mentioned in a 3b1b video somewhere else in this thread) is a great example. Given this was written before advent of LLM, I would love to see someone throw openAI/Anthropic levels of compute at a similar problem to see what the swarm can produce.

https://arxiv.org/html/2404.09937v1

Worth pointing out that PNG is also just awful and should probably die like BMP died. WebP was already available at the time this article was written:

PNG: 1039993 bytes - baseline

optimized*: 541539

WebP: 398974 - lossless mode, of course.

JPEG XL: 310778 - lossless, quick, and easy.

*after monumental compute effort; still probably some room for improvement but the effort is ridiculous.

H.264 High Profile is close to Patent Free. [1] I often wonder how much more efficiency could we get if we were to increase block size outside of spec.

[1] https://meta.wikimedia.org/wiki/Have_the_patents_for_H.264_M...

I've shared this article many times. I work in space and the vast majority of the time, new people insist that all images should be raw. After all, you go to great expense to get somewhere in the name of science, you don't want to discard any information. And that's true in some cases but typically not for a whole mission or even a whole image if you can crop a specific part. Usually 100 or 1000 images instead of one is more valuable even if the compression is lossy.

That's just image compression though. Very few space-grade cameras support video compression. In a lot of cases it works well even though you think you're after a series of still images.

This reminds me that I still haven't watched `part 2` of a very nice video from 3Blue1Brown - "Compression is Intelligence". YouTube suggestions, definitely failed me with this one.

One year left before the "last major patent" expires.

I remember reading this article when it was new, and I left fully agreeing, but more broadly: modern video compression is magic.

Like, it's baffling to me how much we can compress high-quality video nowadays. HEVC and AV1 and VP9 are all also very cool and they still sort of break my brain. I understand how they work but it doesn't feel like it should compress as well as it does.

I'm surprised that 10 years later, the article still teaches an incorrect conceptual understanding of "quantization".

The context heavily implies that quantization is "discarding information" generally (e.g. clipping, masking). The wikipedia article he links in the footnote disagrees, and so do I:

> quantization is the process of mapping input values from a large set (often a continuous set) to output values in a (countable) smaller set, often with a finite number of elements.

Quantization is a function over values, or the image itself. Clipping and masking, by contrast, are functions over the space/frequency dimensions, i.e. the pixel indices ("preimage"). A rule like "apply a circular mask" or "drop the first quadrant" does not depend on the values!

The most extreme example of quantization is a threshold filter. Regardless of location, output "1" if the pixel is brighter than X, or "0" otherwise. This depends only on the value, not on the location. He also misses the opportunity to point out that chroma subsampling is another application of quantization, this time to chroma values (drop the low-order bits).

Codecs like H.264 and JPEG use spatially-varying quantization rules to effectively discard more information at high frequencies, so his explanation of H.264 is spot on, but a non-expert encountering quantization for the first time would probably come away with the wrong idea. For an article explicitly targeted to non-experts that goes out of its way to introduce the term, it's a pretty important oversight.

undefined

This reminds me of that old nursery rhyme. Billy was a chemist's son, but Billy is no more, what he thought was H2O was H.264

Ref.:

https://news.ycombinator.com/item?id=38099633

Article is dated (2016).

I wonder if an AI can make a better codec, and if we would think of it as "magic".

It is tradition to re-read this every year. Along with the m1 blackmagic one.

Socrates was certainly a pioneer of conveying information efficiently

Well, that was disappointing. Everything in that article is generic to any video compression codec, even MPEG. I thought this would discuss why H.264 compresses so much more than predecessors, but not a word about that. Should have been titled "video compression is magic".

The only question I have:

Should I use H.264 or H.265?

[flagged]

"The H.264 video, 300 frames long is 175KB. A single frame of that video in PNG is 1015KB. It looks like we're storing 300 times the amount of data in the video. But the file size is a fifth. So H.264 would seem to be 1500x as efficient as PNG. How is this even possible? All right, what's the trick?"

Mostly the trick is they're comparing lossy to lossless compression which doesn't even make any sense.