Nvidia HEVC problems

[Jiehfeng]

Hi, I was told to post here on this forum.

I am having problems with using Nvidia HEVC (I have an RTX 3080). When I use the default settings, I get a very small video file (about 20MB) and the quality is extremely poor. (the original video is about 1GB and 26.7Mbps)

This is the default:


I followed the advice of someone on another forum and set the settings as so:



But this time when I export, it exports immediately and the file is a broken 258 byte file. What am I doing wrong?

[eumagga0x2a]

Does "Constant Quality" (actually, constant qp) mode work? Could you please provide admlog.txt located at %localappdata%\avidemux\ from starting Avidemux, loading the source video, trying to encode a short portion of it using the default HQ preset and then the Variable Bitrate mode?

I'm at disadvantage here as my NVIDIA graphics card doesn't support HEVC at all (not even decoding). I could imagine that the latest and greatest hardware + maybe the latest and greatest graphics drivers need the latest and greatest nv-codec-headers version or even newer FFmpeg libraries than currently used in Avidemux. I hope that the log might contain clues why encoding doesn't work right.

Please ensure you use the latest available nightly build --> https://avidemux.org/nightly/win64/

[Jiehfeng]

Does "Constant Quality" (actually, constant qp) mode work? Could you please provide admlog.txt located at %localappdata%\avidemux\ from starting Avidemux, loading the source video, trying to encode a short portion of it using the default HQ preset and then the Variable Bitrate mode?

I'm at disadvantage here as my NVIDIA graphics card doesn't support HEVC at all (not even decoding). I could imagine that the latest and greatest hardware + maybe the latest and greatest graphics drivers need the latest and greatest nv-codec-headers version or even newer FFmpeg libraries than currently used in Avidemux. I hope that the log might contain clues why encoding doesn't work right.

Please ensure you use the latest available nightly build --> https://avidemux.org/nightly/win64/

Got the latest nightly build like you said, for some reason the installer didn't uninstall my current Avidemux installation, I'll have to uninstall it separately.

I did as you said, same two errors, the log should be attached to this post.

[eumagga0x2a]

Regarding uninstaller, you probably had a VC++ build like all recent release builds. They can coexist with MinGW builds like the latest nightly from the win64 subdirectory, just don't run them simultaneously.

Thank you for the log, I'll try to look into it tonight. There are no relevant errors, however (libavcodec does not protest, everything — apart from bitrate in dual-pass mode, maybe — works, the result is bad). Does encoding with NVENC to H.264 work fine?

[Jiehfeng]

Regarding uninstaller, you probably had a VC++ build like all recent release builds. They can coexist with MinGW builds like the latest nightly from the win64 subdirectory, just don't run them simultaneously.

Thank you for the log, I'll try to look into it tonight. There are no relevant errors, however (libavcodec does not protest, everything — apart from bitrate in dual-pass mode, maybe — works, the result is bad). Does encoding with NVENC to H.264 work fine?

Ah I see, that makes sense.

If you mean Nvidia H264, that works perfectly on stock settings at least.

EDIT: If you meant an NVENC video, I just recorded one with OBS, and I converted it to Nvidia H264 with Avidemux and it worked perfectly.

[eumagga0x2a]

If you mean Nvidia H264, that works perfectly on stock settings at least.

I see, thanks. This means, NVENC in general works, but not the HEVC encoder, at least not in the default mode and not in the dual-pass VBR mode. Is there a mode which works, maybe the constant bitrate mode (not really desireble, I know)?

I wonder whether the problem can be a driver issue. https://github.com/moonlight-stream/moonlight-qt/issues/447 lets me think that there is some breakage on NVIDIA's end (NVIDIA doesn't list any official drivers for Windows 8 anyway, albeit there is one for Windows 7 , but I don't think this matters here).

[Jiehfeng]

If you mean Nvidia H264, that works perfectly on stock settings at least.

I see, thanks. This means, NVENC in general works, but not the HEVC encoder, at least not in the default mode and not in the dual-pass VBR mode. Is there a mode which works, maybe the constant bitrate mode (not really desireble, I know)?

I wonder whether the problem can be a driver issue. https://github.com/moonlight-stream/moonlight-qt/issues/447 lets me think that there is some breakage on NVIDIA's end (NVIDIA doesn't list any official drivers for Windows 8 anyway, albeit there is one for Windows 7 , but I don't think this matters here).

Just tried constant bitrate mode, horrible quality (about 800kbps was the result). Constant quality mode kind of works though, I set it to 0 and even though the quality is decent, it is still much less than the original bitrate (the result was 7187kbps while the original was 26944 (same original file used throughout the testing in this thread)).

Hmm could be, I'll try Handbrake to see if I can rule it out.

EDIT: Tried Handbrake, similar story. In Nvidia H.265 (I assume that means HEVC), I chose average bitrate of 26944kbps and the result while decent, the bitrate is lower than what I set. Even if I chose constant quality and set that to 0, the result was about 11000kbps. Just thought It'd provide a better picture of this situation.

[eumagga0x2a]

Which version of NVIDIA drivers is installed on your system? The graphics card is very new, there are not many driver versions which support it. I wonder whether the oldest one – 456.38 – already exhibits the issue.

[Jiehfeng]

Which version of NVIDIA drivers is installed on your system? The graphics card is very new, there are not many driver versions which support it. I wonder whether the oldest one – 456.38 – already exhibits the issue.

Not the very latest but relatively recent, 456.55.

[eumagga0x2a]

Could you please try going back to 456.38?

[Jiehfeng]

Could you please try going back to 456.38?

Sorry, I don't want to risk it. There was a crashing issue that was fixed with my driver so I don't wanna go back to an old driver.

[eumagga0x2a]

Sure, I understand. You will have to wait for NVIDIA to sort things out then.

By the way, do you have a Linux system installed as dual-boot on the same hardware to test whether the issue is cross-platform?

[Jiehfeng]

Sure, I understand. You will have to wait for NVIDIA to sort things out then.

By the way, do you have a Linux system installed as dual-boot on the same hardware to test whether the issue is cross-platform?

No, but should I try that? Will it help you?
I tried Adobe Premiere Pro and its HEVC option, the exported video has 7000kbps bitrate while the original is 23000kbps, maybe HEVC's nature is to lower the bitrate? The quality looks almost identical to me either way.

[eumagga0x2a]

No, but should I try that?

No, please don't bother then, too much of a hassle.

I tried Adobe Premiere Pro and its HEVC option, the exported video has 7000kbps bitrate while the original is 23000kbps, maybe HEVC's nature is to lower the bitrate? The quality looks almost identical to me either way.

Does Premiere use the NVENC hardware encoder or perform encoding on the CPU? Software encoders are not expected to be affected by the NVENC issue.

Regarding lower bitrate: of course, this is the purpose of HEVC to provide a quality similar to H.264 at a much lower bitrate (3x lower is perfectly possible depending on source). The problem with NVENC was the specified quality not respected.

[Jiehfeng]

No, but should I try that?

No, please don't bother then, too much of a hassle.

I tried Adobe Premiere Pro and its HEVC option, the exported video has 7000kbps bitrate while the original is 23000kbps, maybe HEVC's nature is to lower the bitrate? The quality looks almost identical to me either way.

Does Premiere use the NVENC hardware encoder or perform encoding on the CPU? Software encoders are not expected to be affected by the NVENC issue.

Regarding lower bitrate: of course, this is the purpose of HEVC to provide a quality similar to H.264 at a much lower bitrate (3x lower is perfectly possible depending on source). The problem with NVENC was the specified quality not respected.

I chose CUDA acceleration when I first started the project, but when exporting, selecting HEVC has it locked to "Software Encoding" and I can't change it. I think the GPU is utilised but I guess it's still Software Encoding according to you.

Ah I see, by quality not respected, is it a bad enough difference to not choose HEVC? It looks almost the same to me.

What about Handbrake? Though the GPU load was low at 4%, it wasn't at 0% which is what it was at when the rendering finished. Does it also use Software Encoding for HEVC? The output was fine.

EDIT: Also, in Premiere Pro the GPU load was over 10%.