x265 to x264 looks so different

[Pipalo]

Hi, I'm trying to compress an x265 video to a lighter x264 one.
In Avidemux viewport, the video is quite different already.
What I get with the rendered file is exactly the same.
With 2.7.8 the video is too dull.
With 2.8.0 r220103 the video shows too much brightness and contrast.

Here is an excerpt from MediaInfo:

Video
ID : 1
Format: HEVC
Format/Info: High Efficiency Video Coding
Format profile: Main 10@L5.1@Main
HDR format: SMPTE ST 2086, HDR10 compatible
Codec ID: V_MPEGH/ISO/HEVC
Duration: 1 h 02 min
Bit rate: 16.2 Mb/s
Width: 3 840 pixels
Height: 1 608 pixels
Display aspect ratio: 2.40:1
Frame rate mode: Constant
Frame rate: 23.976 (24000/1001) FPS
Color space: YUV
Chroma subsampling: 4:2:0 (Type 2)
Bit depth: 10 bits
Bits/(Pixel*Frame): 0.110
Stream size: 29.8 GiB (98%)
Writing library: x265 3.4+22-772b3229d157:[Linux][GCC 7.5.0][64 bit] 10bit
...
Default: Yes
Forced: No
Color range: Limited
Color primaries: BT.2020
Transfer characteristics: PQ
Matrix coefficients: BT.2020 non-constant
Mastering display color pri: Display P3
Mastering display luminance: min: 0.0050 cd/m2, max: 4000 cd/m2
Maximum Content Light Level: 636 cd/m2
Maximum Frame-Average Light: 103 cd/m2

I suppose the 10bits and HDR stuff are no stranger to my problem.
I'm on windows 10, I use mpc-hc64 with no HDR screen.
My question:
Is there a way to find the correct colors back in the output video?

Two other questions slightly related to this issue:
- The file is rather big, when imported, avidemux does not create an idx2 file.
So it takes some time each time... Any way to change that ?
- When I try to render a short sample, with the same x264 settings, 2.7.8 renders a smaller file, 2.8 renders a file bigger than the original.
Is that normal ?

[szlldm]

With 2.7.8 the video is too dull.

It is because 2.7.8 does not have HDR tonemapping.

You can tweak settings in the Video/HDR tone mapping menu.

[eumagga0x2a]

Alternatively, you could disable HDR tone mapping but instead configure matching VUI (video usability information) in the configuration of the x264 encoder plugin. The re-encoded video would look dull in Avidemux, but video players like mpv with "gpu" video output working should be able to display it with sort of correct looking colors. Tone mapping is performed by the graphics card driver then.

[eumagga0x2a]

- The file is rather big, when imported, avidemux does not create an idx2 file.
So it takes some time each time... Any way to change that ?

Dumping memory-only index for MKV files to disk was one of my secret wishes from early days as I absolutely could not code :-) Maybe I'll return to this idea in the future (or szlldm will pick it up as he is hundreds of times faster).

By all means, it is really worth to find out why we are that incredibly slow when indexing cluster. The next step probably cannot be made faster, increasing pressure to preserve index once created.

[szlldm]

When I try to render a short sample, with the same x264 settings, 2.7.8 renders a smaller file, 2.8 renders a file bigger than the original.
Is that normal ?

Without HDR tonemapping pixels are truncated to 8 bit, so there is information and precision loss (exactly why video looks dull), and therefore it could be more compressible.

[eumagga0x2a]

By all means, it is really worth to find out why we are that incredibly slow when indexing cluster.

It turns out to be that "incredibly slow" is just the reading speed of the mass storage device. If the file is small enough to fit into memory, opening it the second time a short while after having loaded it for the first time, cluster indexing is finished almost instantly, at least on Linux.

So no wrongdoing on part of Avidemux here.

Regarding output file size HEVC --> H.264, a file smaller than the original means the quality must be really poor (or the video needs to be downscaled).

[Pipalo]

Thanks guys for the great help,
and thanks for the knowledge.

Tweaking the tonemapping menu now.

Have a nice day