x264 advanced settings encoding - questions and experiments on how to reduce banding

[Rafael Defavari]

Hi guys,

I am new to this forum and also new to the art of encoding. I am having a severe problem with banding. I know I am not the first one and I have already read lots of posts about this and how to fight it. I would like your help to see if I can add anything to my workflow to improve the video quality or if I am doing something wrong here.

The source file is a 10bit avi v210 file with no visible banding (the sequence is 04min 44s long and has size of 46Gb) but I’ll show only the most problematic scene. Here’s a screenshot showing how it looks like in the 10bit source file (the picture has been stretched so it can exagerate the fading):

http://i.imgur.com/NQf0g7F.jpg

The first thing I made with this file was making it 8bit with dither loading a .avs file into Virtualdub using the following code:

LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\plugins\ffms2.dll")
FFvideosource("D:\Arq\encoding\timelapse 30fps master 10bit.avi",enable10bithack=true)
f3kdb_dither()

Then, the output is a 8bit avi (52Gb size) with a zero (or very very little and way acceptable) banding visible as you can see in this screenshot:

http://i.imgur.com/DlTdvae.jpg

So I used HandBrake 0.9.9.5530 - 64 bit version to encode a .mp4 file via x264. One important thing to note is: I am aiming the best quality possible that I can output for a 5Gb file. 5Gb is the largest size Vimeo will accept my video so that`s my limit. I don`t mind if the file takes 2 weeks to encode to gain just one pixel better. The video will be online for years to come so encoding time is definetly out of question. The important thing is the quality above everything that can fit this size.

Settings:

Picture - Anamorphic strict (1920x1080), custom croppping (0 for top, left, right and bottom).
Filters - Detelecine default, Ddecomb default. Deinterlace, Denoise and Deblock Off.
Video - Codec x264. Framerate constant (30fps in the source and the output). 2-pass encoding with average bitrate 145000 (that will give me a file around 4.9Gb).
Advanced - ref=16:bframes=16:b-adapt=2:direct=auto:me=tesa:subme=11:merange=64:analyse=all:trellis=2:deblock=-1,-1

Here`s a screenshot of the advanced tab on Handbrake:

http://i.imgur.com/cDYsuLD.jpg

This settings takes around 12h to encode. Fine for me.
This is the output as seen from virtualdub:

http://i.imgur.com/JJ6dU6U.jpg

And this is the same file but played with Media Classic Player:

http://i.imgur.com/VHBlX0N.jpg

When I play the 8bit dithered uncompressed file on Media Player Classic, the output range is 0-255. x264 changed the output it to 16-235.
Question #1: Is this normal?

I had to set the output range on MPC to 16-235 to match virtualdub colors. But on virtualdub absolute black is darker.
Question #2: Is Virtualdub transforming my video file in 0-255 to preview it?

Question #3: Does Vimeo do the same, i.e. expect a 16-235 and transform into 0-235? I had problems with gamma shift before, when uploading to vimeo, so it`s an important thing to consider. Is x264 outputting a 16-235 file the correct workflow for vimeo? (PS. I have read vimeo guidelines, and there`s no metion for the output range)

The MPC screenshot above is just an extra: all of the screenshots below will be made using Virtualdub so we can compare with the 10bit file.

As you can see, obvious and strong banding after x264, even with dither. The problem seem to be on x264. So I started reading on which values to tweak to try to improve this. I have read a lot and have made a few experiments.

Firstly, I changed AQ Streght to 0.8 and add the no-mbtree parameter. Here`s a sreenshot on the output file with these changes:

http://i.imgur.com/0dC6Kzy.jpg

Not a change. And then I also set deblock to 3,3, used the No-DCT-Decimate with the AQ strenght 0,8 and no-mbtree. The result is:

http://i.imgur.com/2zGc7WZ.jpg

Not a change also.

Question #4: What I am doing wrong here? Anyhow I can tweak x264 to make this fading better with less banding?

I don`t know what else I can do to reduce this. I don`t think adding noise to the source file will help, since the 8bit source has already no banding.

 

[paulherrin]

short answer for #4... no, i don't think so. even your highest quality 8-bit file has banding. 8-bit is not enough for such situations, period. you've got a smooth gradation of luminance and complementary colors blending and a lot of contrast, local and global.

do keep in mind on vimeo you can replace videos, so when there is better quality encoding for web playback you can swap it out. also, you can upgrade your account so at least to have higher file sizes...

 

[Rafael Defavari]

short answer for #4... no, i don't think so. even your highest quality 8-bit file has banding. 8-bit is not enough for such situations, period. you've got a smooth gradation of luminance and complementary colors blending and a lot of contrast, local and global.

do keep in mind on vimeo you can replace videos, so when there is better quality encoding for web playback you can swap it out. also, you can upgrade your account so at least to have higher file sizes...

Hi Paul, thanks for the input. I don't know if I can upload a 10bit file to vimeo, but even if I can, vimeo will encode it to 8 bit anyway. So, it's better use dither myself than vimeo using nothing.

I agree that even the 8 bit uncompressed has banding. But at that level is fine for me. The problem is that after x264 the banding is way more pronounced. I just want to get closer to the 8bit uncompressed. At 145kbs shouldn't be that much different.

 

[paulherrin]

exactly, and there aren't many screens out there unfortunately either... hopefully that's the next frontier after UHD.

anyways, look - your problem there plain and simple is that you still need
1. a more efficient codec (hevc/x265) which is on the way... (as i said you can replace)
2. a higher bitrate. and that's what the pricey pro plan is for.

i'm not saying that encoding isn't an art unto itself, one that i admit i've not mastered. but it's also a major moving target, especially right this second. but the fact is that you've got a limited amount of data per frame, and the codec is responding accordingly. you're not going to match a 5gb x264 to 50gb avi - sorry :/

now, it might be possible to add a bunch of noise/dithering to particular shots - and yes you might want to do that before compression.

the upside is that all things are even on the web (well, unless you have a pro plan or host your own streaming files) and everybody is used to seeing compression artifacts online... so it's actually less distracting to other people than it might seem to you (i think). but yeah, web compression sucks, but it is what it is and it will improve as we go on. hopefully screens will, too, and we'll eventually get far greater dynamic range.

 

[Rafael Defavari]

Hi guys, thanks a lot for all the help given. I have finished the project and if you wish to take a look at the result you can click here to be redirected to the showcase thread. Opinions are very welcome. Thanks!

 

[Alexander Mejia]

Avisynth is so wonderful because you can script your plugins to happen in any order.

I would personally convert to 8-bpc and 4:2:0 before you go into adding static dithering to your image. It makes more sense to add the pattern to your footage before you start reducing resolution of the cB and cR images.