Image quality problem when encoding from dvd to X264 .mkv files

[Zapata]

I've been using Avidemux for more than 15 years now, and I really like it since I've started with it to encode dvb TS files among others.

But I am now experiencing quality problems when encoding from dvd vob files to mkv with x264 video.

I may add that I still use Avidemux to encode DVB H264 .ts files to lighter MKV x264 files using the recommended settings and I usually don't have quality problems with them.

But when I tried to do the same with some dvd files I had to increase considerably the bitrate in order to get an "acceptable" video quality, even with higher quality x264 settings.

So I finally tried encoding with Handbrake, and I "unfortunately" managed to get a better quality using SLOW preset, no extra settings, and even a lower bitrate.

That's what make me sad, since I had always been satisfied with Avidemux as you understand.

I may provide you sample encoded files with Avidemux and Handbrake, the x264 settings I used, or at least file information to help find an explanation.

[eumagga0x2a]

I may add that I still use Avidemux to encode DVB H264 .ts files to lighter MKV x264

I would strongly recommend against this procedure unless you need to do some filtering (e.g. adding a logo, blurring an area of the picture, increasing contrast and so on). Any re-encoding with a lossy codec reduces quality. At least using a superior codec like HEVC might give the unavoidable quality loss some justification by reducing bitrate and thus file size. But H.264??

But when I tried to do the same with some dvd files I had to increase considerably the bitrate in order to get an "acceptable" video quality, even with higher quality x264 settings.

The first question would be whether the DVD source is interlaced or not. If it is interlaced, then you absolutely must either insert a deinterlacer (on Windows is yadif probably the best option available in Avidemux) as the first filter in the filter chain or, if you don't want to apply any filters, tell x264 that the source is interlaced and which field is first.

Of course, I presume that you have disabled any post-processing in Avidemux. Post-processing is skipped when video is decoded in hardware (on Windows: via DXVA2) which may be the case for H.264 but active when video is decoded purely in software which would be the case for MPEG-2.

I further assume that you use a supported version of Avidemux which is at least the last 2.8.1 release.

[Zapata]

Thank you for reply. About the first point,

I would strongly recommend against this procedure unless you need to do some filtering (e.g. adding a logo, blurring an area of the picture, increasing contrast and so on). Any re-encoding with a lossy codec reduces quality. At least using a superior codec like HEVC might give the unavoidable quality loss some justification by reducing bitrate and thus file size. But H.264??

, I can explain my view : I know that encoding an H264 file again with H264 will result in a quality loss, but I do my best in order to get a re-encoded file without visible or at least very subtle quality loss, so that I can divide the size by 2 or 3, sometimes even 4.

Then, with respect to DVD re-encoding, I first will keep in mind to check whether the video is interlaced, and I've also disabled Post-Processing which allowed me to get a good quality when encoding with x264 ( with a bitrate range from 1200 to 1700 kbps most of the time).

[eumagga0x2a]

I know that encoding an H264 file again with H264 will result in a quality loss, but I do my best in order to get a re-encoded file without visible or at least very subtle quality loss, so that I can divide the size by 2 or 3, sometimes even 4.

I would rule out that recompressing a DVB (even DVB-S) source with x264 into H.264 would be able to more than halve the bitrate without a major loss of quality (unless you either reduce the resolution, reduce the frame rate, reduce sharpness or apply any combination of the above).

However, depending on the tool used to capture the broadcast, the original MPEG-TS file may contain a lot of additional data like other video streams sharing the same transponder or empty packets for padding as the transponder bitrate has to stay constant. Just saving such a file in copy mode to an MKV or MP4 (if audio tracks are compatible) using a supported Avidemux version would discard additonal video tracks as well as padding which may significantly reduce the overall file size.

[Zapata]

Sorry for taking time to reply. As you said,

I would rule out that recompressing a DVB (even DVB-S) source with x264 into H.264 would be able to more than halve the bitrate without a major loss of quality (unless you either reduce the resolution, reduce the frame rate, reduce sharpness or apply any combination of the above).

, I can reduce the size of DVB-T recorded files with a 4000 to 7000 kbps bitrate, a resolution of 1920*1080 pixels, and 50 fps to 2-4 times smaller re-compressed files with 1100 to 1800 kbps bitrate, a 576 vertical pixels resolution (1024*576 for 16/9 aspect ratio) and a frame rate converted from 50 to 25 fps.

[eumagga0x2a]

I can reduce the size of DVB-T recorded files with a 4000 to 7000 kbps bitrate, a resolution of 1920*1080 pixels, and 50 fps to 2-4 times smaller re-compressed files with 1100 to 1800 kbps bitrate, a 576 vertical pixels resolution (1024*576 for 16/9 aspect ratio) and a frame rate converted from 50 to 25 fps.

Downscaling of the picture and halving of the frame rate constitute a huge loss of quality, so no wonder x264 is able to produce output which takes less disk space than the source.

[Zapata]

I understand that this is a theoritical/technical loss of quality, but it allows me to keep a good visual quality and to save a lot of disk space ...
So I will keep processing that way, and maybe start using x265 one day.

Thank you for Avidemux, and your quick replies.

[Geo_log]

But I am now experiencing quality problems when encoding ... with x264 video.
...
So I finally tried encoding with Handbrake, and I "unfortunately" managed to get a better quality

I can confirm this. 
Dramatic loss of quality with AviDemux v.2.8.1 when re-encoding ordinary x264 video file with ordinary AviDemux x264 settings: NOT Use advanced configuration, Preset: slow, Tuning: none, Profile: High, Encoding Mode: Constant Rate Factor (Single Pass), Quality: 19.  Dramatic - when compare to the same settings re-encode with FFMpeg or StaxRip. Even size of v.2.8.1-prodiced file smaller than FFMpeg's and StaxRip's (which the same).
But.
When I downgraded to AviDemux v.2.8.0, the quality and size became the same as with FFMpeg.

P.S. Only v.2.8.0 (and previous) is good. Even avidemux_r220207_win64Qt5_130.zip has this problem.
I can provide my video files and settings.

[eumagga0x2a]

Dramatic loss of quality with AviDemux v.2.8.1 when re-encoding ordinary x264 video file with ordinary AviDemux x264 settings

Please make sure post-processing in Avidemux is completely disabled, active post-processing is the usual reason for poor quality when re-encoding video in Avidemux, no matter which encoder.

P.S. Only v.2.8.0 (and previous) is good.

Most likely post-processing in old versions was simply broken (or inactive with the video decoder used in your setup).

[Geo_log]

Please make sure post-processing in Avidemux is completely disabled

Thank you very much, that solved the problem.

And many thanks for the wonderful AviDemux!

[Cormy1]

Please make sure post-processing in Avidemux is completely disabled, active post-processing is the usual reason for poor quality when re-encoding video in Avidemux, no matter which encoder.

Does this imply the default settings of horizontal and vertical deblocking are NEVER recommended?
Are there times when they are desired? Why are they on by default?

[eumagga0x2a]

They are off by default. But if user config says otherwise, Avidemux doesn't override them.