Avidemux Forum

Hi, I was told to post here on this forum.

I am having problems with using Nvidia HEVC (I have an RTX 3080). When I use the default settings, I get a very small video file (about 20MB) and the quality is extremely poor. (the original video is about 1GB and 26.7Mbps)

This is the default:


I followed the advice of someone on another forum and set the settings as so:



But this time when I export, it exports immediately and the file is a broken 258 byte file. What am I doing wrong?

Does "Constant Quality" (actually, constant qp) mode work? Could you please provide admlog.txt located at %localappdata%\avidemux\ from starting Avidemux, loading the source video, trying to encode a short portion of it using the default HQ preset and then the Variable Bitrate mode?

I'm at disadvantage here as my NVIDIA graphics card doesn't support HEVC at all (not even decoding). I could imagine that the latest and greatest hardware + maybe the latest and greatest graphics drivers need the latest and greatest nv-codec-headers version or even newer FFmpeg libraries than currently used in Avidemux. I hope that the log might contain clues why encoding doesn't work right.

Please ensure you use the latest available nightly build --> https://avidemux.org/nightly/win64/

Quote from: eumagga0x2a on November 24, 2020, 08:14:06 PMDoes "Constant Quality" (actually, constant qp) mode work? Could you please provide admlog.txt located at %localappdata%\avidemux\ from starting Avidemux, loading the source video, trying to encode a short portion of it using the default HQ preset and then the Variable Bitrate mode?

I'm at disadvantage here as my NVIDIA graphics card doesn't support HEVC at all (not even decoding). I could imagine that the latest and greatest hardware + maybe the latest and greatest graphics drivers need the latest and greatest nv-codec-headers version or even newer FFmpeg libraries than currently used in Avidemux. I hope that the log might contain clues why encoding doesn't work right.

Please ensure you use the latest available nightly build --> https://avidemux.org/nightly/win64/

Got the latest nightly build like you said, for some reason the installer didn't uninstall my current Avidemux installation, I'll have to uninstall it separately.

I did as you said, same two errors, the log should be attached to this post.

Regarding uninstaller, you probably had a VC++ build like all recent release builds. They can coexist with MinGW builds like the latest nightly from the win64 subdirectory, just don't run them simultaneously.

Thank you for the log, I'll try to look into it tonight. There are no relevant errors, however (libavcodec does not protest, everything — apart from bitrate in dual-pass mode, maybe — works, the result is bad). Does encoding with NVENC to H.264 work fine?

Quote from: eumagga0x2a on November 25, 2020, 08:10:47 AMRegarding uninstaller, you probably had a VC++ build like all recent release builds. They can coexist with MinGW builds like the latest nightly from the win64 subdirectory, just don't run them simultaneously.

Thank you for the log, I'll try to look into it tonight. There are no relevant errors, however (libavcodec does not protest, everything — apart from bitrate in dual-pass mode, maybe — works, the result is bad). Does encoding with NVENC to H.264 work fine?

Ah I see, that makes sense.

If you mean Nvidia H264, that works perfectly on stock settings at least.

EDIT: If you meant an NVENC video, I just recorded one with OBS, and I converted it to Nvidia H264 with Avidemux and it worked perfectly.

Quote from: Jiehfeng on November 25, 2020, 08:56:53 AMIf you mean Nvidia H264, that works perfectly on stock settings at least.

I see, thanks. This means, NVENC in general works, but not the HEVC encoder, at least not in the default mode and not in the dual-pass VBR mode. Is there a mode which works, maybe the constant bitrate mode (not really desireble, I know)?

I wonder whether the problem can be a driver issue. https://github.com/moonlight-stream/moonlight-qt/issues/447 lets me think that there is some breakage on NVIDIA's end (NVIDIA doesn't list any official drivers for Windows 8 anyway, albeit there is one for Windows 7 ???, but I don't think this matters here).

Quote from: eumagga0x2a on November 25, 2020, 10:31:45 PM

Quote from: Jiehfeng on November 25, 2020, 08:56:53 AMIf you mean Nvidia H264, that works perfectly on stock settings at least.

I see, thanks. This means, NVENC in general works, but not the HEVC encoder, at least not in the default mode and not in the dual-pass VBR mode. Is there a mode which works, maybe the constant bitrate mode (not really desireble, I know)?

I wonder whether the problem can be a driver issue. https://github.com/moonlight-stream/moonlight-qt/issues/447 lets me think that there is some breakage on NVIDIA's end (NVIDIA doesn't list any official drivers for Windows 8 anyway, albeit there is one for Windows 7 ???, but I don't think this matters here).

Just tried constant bitrate mode, horrible quality (about 800kbps was the result). Constant quality mode kind of works though, I set it to 0 and even though the quality is decent, it is still much less than the original bitrate (the result was 7187kbps while the original was 26944 (same original file used throughout the testing in this thread)).

Hmm could be, I'll try Handbrake to see if I can rule it out.

EDIT: Tried Handbrake, similar story. In Nvidia H.265 (I assume that means HEVC), I chose average bitrate of 26944kbps and the result while decent, the bitrate is lower than what I set. Even if I chose constant quality and set that to 0, the result was about 11000kbps. Just thought It'd provide a better picture of this situation.

Which version of NVIDIA drivers is installed on your system? The graphics card is very new, there are not many driver versions which support it. I wonder whether the oldest one – 456.38 – already exhibits the issue.

Quote from: eumagga0x2a on November 26, 2020, 10:13:45 PMWhich version of NVIDIA drivers is installed on your system? The graphics card is very new, there are not many driver versions which support it. I wonder whether the oldest one – 456.38 – already exhibits the issue.

Not the very latest but relatively recent, 456.55.

Could you please try going back to 456.38?

Quote from: eumagga0x2a on November 27, 2020, 05:08:25 PMCould you please try going back to 456.38?

Sorry, I don't want to risk it. There was a crashing issue that was fixed with my driver so I don't wanna go back to an old driver.

Sure, I understand. You will have to wait for NVIDIA to sort things out then.

By the way, do you have a Linux system installed as dual-boot on the same hardware to test whether the issue is cross-platform?

Quote from: eumagga0x2a on November 28, 2020, 02:00:08 PMSure, I understand. You will have to wait for NVIDIA to sort things out then.

By the way, do you have a Linux system installed as dual-boot on the same hardware to test whether the issue is cross-platform?

No, but should I try that? Will it help you?
I tried Adobe Premiere Pro and its HEVC option, the exported video has 7000kbps bitrate while the original is 23000kbps, maybe HEVC's nature is to lower the bitrate? The quality looks almost identical to me either way.

Quote from: Jiehfeng on November 28, 2020, 02:24:05 PMNo, but should I try that?

No, please don't bother then, too much of a hassle.

Quote from: Jiehfeng on November 28, 2020, 02:24:05 PMI tried Adobe Premiere Pro and its HEVC option, the exported video has 7000kbps bitrate while the original is 23000kbps, maybe HEVC's nature is to lower the bitrate? The quality looks almost identical to me either way.

Does Premiere use the NVENC hardware encoder or perform encoding on the CPU? Software encoders are not expected to be affected by the NVENC issue.

Regarding lower bitrate: of course, this is the purpose of HEVC to provide a quality similar to H.264 at a much lower bitrate (3x lower is perfectly possible depending on source). The problem with NVENC was the specified quality not respected.

Quote from: eumagga0x2a on November 28, 2020, 03:06:54 PM

Quote from: Jiehfeng on November 28, 2020, 02:24:05 PMNo, but should I try that?

No, please don't bother then, too much of a hassle.

Quote from: Jiehfeng on November 28, 2020, 02:24:05 PMI tried Adobe Premiere Pro and its HEVC option, the exported video has 7000kbps bitrate while the original is 23000kbps, maybe HEVC's nature is to lower the bitrate? The quality looks almost identical to me either way.

Does Premiere use the NVENC hardware encoder or perform encoding on the CPU? Software encoders are not expected to be affected by the NVENC issue.

Regarding lower bitrate: of course, this is the purpose of HEVC to provide a quality similar to H.264 at a much lower bitrate (3x lower is perfectly possible depending on source). The problem with NVENC was the specified quality not respected.

I chose CUDA acceleration when I first started the project, but when exporting, selecting HEVC has it locked to "Software Encoding" and I can't change it. I think the GPU is utilised but I guess it's still Software Encoding according to you.

Ah I see, by quality not respected, is it a bad enough difference to not choose HEVC? It looks almost the same to me.

What about Handbrake? Though the GPU load was low at 4%, it wasn't at 0% which is what it was at when the rendering finished. Does it also use Software Encoding for HEVC? The output was fine.

EDIT: Also, in Premiere Pro the GPU load was over 10%.

Quote from: Jiehfeng on November 28, 2020, 04:02:59 PMI chose CUDA acceleration when I first started the project, but when exporting, selecting HEVC has it locked to "Software Encoding" and I can't change it. I think the GPU is utilised but I guess it's still Software Encoding according to you.

Looks like Premiere is able to detect that something doesn't work right with the NVIDIA HW accelerated HEVC encoder.

Quote from: Jiehfeng on November 28, 2020, 04:02:59 PMAh I see, by quality not respected, is it a bad enough difference to not choose HEVC?

As the NVENC is apparently broken with Avidemux (with libavcodec as used in Avidemux) on Windows with the NVIDIA graphics card and drivers you use (and maybe with all driver versions available for this card), it is up to you to either select the only software HEVC encoder available (x265, superior by far to NVENC but very expensive computationally) or to choose a different codec alltogether. By choosing H.264, you pay the price by needing a higher bitrate for given quality (but get a better compatibility in return).

Quote from: Jiehfeng on November 28, 2020, 04:02:59 PMWhat about Handbrake? Though the GPU load was low at 4%, it wasn't at 0% which is what it was at when the rendering finished. Does it also use Software Encoding for HEVC? The output was fine.

Can't tell, probably Handbrake is using GPU for decoding only. Encoding a video with bare command-line ffmpeg and hevc_nvenc might provide some clues:

ffmpeg -i "DriveLetter:\Path\To\SourceVideo" -c:v hevc_nvenc -preset:v hq -c:a copy "DriveLetter:\Path\To\OutputVideo.mkv"

Quote from: eumagga0x2a on November 28, 2020, 06:49:18 PM

Quote from: Jiehfeng on November 28, 2020, 04:02:59 PMI chose CUDA acceleration when I first started the project, but when exporting, selecting HEVC has it locked to "Software Encoding" and I can't change it. I think the GPU is utilised but I guess it's still Software Encoding according to you.

Looks like Premiere is able to detect that something doesn't work right with the NVIDIA HW accelerated HEVC encoder.

Quote from: Jiehfeng on November 28, 2020, 04:02:59 PMAh I see, by quality not respected, is it a bad enough difference to not choose HEVC?

As the NVENC is apparently broken with Avidemux (with libavcodec as used in Avidemux) on Windows with the NVIDIA graphics card and drivers you use (and maybe with all driver versions available for this card), it is up to you to either select the only software HEVC encoder available (x265, superior by far to NVENC but very expensive computationally) or to choose a different codec alltogether. By choosing H.264, you pay the price by needing a higher bitrate for given quality (but get a better compatibility in return).

Quote from: Jiehfeng on November 28, 2020, 04:02:59 PMWhat about Handbrake? Though the GPU load was low at 4%, it wasn't at 0% which is what it was at when the rendering finished. Does it also use Software Encoding for HEVC? The output was fine.

Can't tell, probably Handbrake is using GPU for decoding only. Encoding a video with bare command-line ffmpeg and hevc_nvenc might provide some clues:

Code Select Expand

ffmpeg -i "DriveLetter:\Path\To\SourceVideo" -c:v hevc_nvenc -preset:v hq -c:a copy "DriveLetter:\Path\To\OutputVideo.mkv"

I see, thanks for the recommendation.

Also, I think I've been mistaken to look at GPU Load, cause in these new GPUs the encoding work is done in a separate area, so I opened up task manager while I was using ffmpeg as you said and the Video Encode section was at 100% the whole time.

The output has a higher bitrate than avidemux, the quality is decent, but not very clear, however not terrible like how it was with avidemux.



Here's what the CMD looked like:

C:\Users\alain\Documents\ShareX\Tools>ffmpeg -i "E:\SharePlay\The Witcher 3\The Witcher 3 2020.11.02 - 13.58.08.04.DVR.mp4" -c:v hevc_nvenc -preset:v hq -c:a copy "C:\Users\alain\Desktop\Tester\Witcher.mp4"
ffmpeg version git-2020-05-27-8b5ffae Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 9.3.1 (GCC) 20200523
  configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libdav1d --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libsrt --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --disable-w32threads --enable-libmfx --enable-ffnvcodec --enable-cuda-llvm --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth --enable-libopenmpt --enable-amf
  libavutil      56. 49.100 / 56. 49.100
  libavcodec     58. 87.101 / 58. 87.101
  libavformat    58. 43.100 / 58. 43.100
  libavdevice    58.  9.103 / 58.  9.103
  libavfilter     7. 83.100 /  7. 83.100
  libswscale      5.  6.101 /  5.  6.101
  libswresample   3.  6.100 /  3.  6.100
  libpostproc    55.  6.100 / 55.  6.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'E:\SharePlay\The Witcher 3\The Witcher 3 2020.11.02 - 13.58.08.04.DVR.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: isommp42
    creation_time   : 2020-11-02T08:28:11.000000Z
    date            : 2020
  Duration: 00:07:19.27, start: 0.000000, bitrate: 26934 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, smpte170m/smpte170m/bt470m), 2560x1440 [SAR 1:1 DAR 16:9], 26731 kb/s, 60 fps, 60 tbr, 90k tbn, 120 tbc (default)
    Metadata:
      creation_time   : 2020-11-02T08:28:11.000000Z
      handler_name    : VideoHandle
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 196 kb/s (default)
    Metadata:
      creation_time   : 2020-11-02T08:28:11.000000Z
      handler_name    : SoundHandle
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> hevc (hevc_nvenc))
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
Output #0, mp4, to 'C:\Users\alain\Desktop\Tester\Witcher.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: isommp42
    date            : 2020
    encoder         : Lavf58.43.100
    Stream #0:0(und): Video: hevc (hevc_nvenc) (Main) (hev1 / 0x31766568), yuv420p, 2560x1440 [SAR 1:1 DAR 16:9], q=-1--1, 2000 kb/s, 60 fps, 15360 tbn, 60 tbc (default)
    Metadata:
      creation_time   : 2020-11-02T08:28:11.000000Z
      handler_name    : VideoHandle
      encoder         : Lavc58.87.101 hevc_nvenc
    Side data:
      cpb: bitrate max/min/avg: 0/0/2000000 buffer size: 4000000 vbv_delay: N/A
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 196 kb/s (default)
    Metadata:
      creation_time   : 2020-11-02T08:28:11.000000Z
      handler_name    : SoundHandle
frame=26356 fps= 84 q=31.0 Lsize=   75437kB time=00:07:19.25 bitrate=1406.9kbits/s speed= 1.4x
video:64413kB audio:10537kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.650333%

C:\Users\alain\Documents\ShareX\Tools>

Thank you, this means that NVENC actually works for HEVC, but not the usual rate control settings (the output bitrate is very low and quantizer is quite high for a "high quality" preset).

Could you please continue testing with ffmpeg which modes produce usable results and which don't? Does it help to specify a fixed, high enough bitrate?

I also suspect that dual-pass mode is broken (is the stats file really populated or empty?).

ffmpeg -h encoder=hevc_nvenc
lists all available options.

PS: Would you please quote only the parts of the text you directly refer to, instead of a complete reply? Thanks!

Quote from: eumagga0x2a on November 29, 2020, 02:16:38 PMThank you, this means that NVENC actually works for HEVC, but not the usual rate control settings (the output bitrate is very low and quantizer is quite high for a "high quality" preset).

I see, great!

Quote from: eumagga0x2a on November 29, 2020, 02:16:38 PMCould you please continue testing with ffmpeg which modes produce usable results and which don't? Does it help to specify a fixed, high enough bitrate?

I tried the lossless option, bitrate is much higher at 59000kbps, looks more or less the same as the source.

Then I tried Constant Bitrate Mode (the regular one, not the hq one) with the same bitrate as the source, the resulting bitrate is lower (as you said that is expected), but the quality is great, almost like the original:



cbr hq also gave similar results, but slightly lower bitrate at 11000kbps.

vbr hq also gave more or less the same result and bitrate.

Quote from: eumagga0x2a on November 29, 2020, 02:16:38 PMI also suspect that dual-pass mode is broken (is the stats file really populated or empty?).

Tried that as well, used the slow preset which says it uses 2 pass, the -rc 2 pass seems to be deprecated. I specified the bitrate to the same as the source here as well. Bitrate was around 11000kbps and quality is more or less the same as the source.

So all in all, all the modes I tried gave similar results and all of them look the same to me and to the source. So does this mean avidemux needs a fix somewhere?

Quote from: Jiehfeng on November 29, 2020, 04:03:20 PMTried that as well, used the slow preset which says it uses 2 pass, the -rc 2 pass seems to be deprecated.

If there was no stats file, it wasn't a 2-pass mode.

Quote from: Jiehfeng on November 29, 2020, 04:03:20 PMSo does this mean avidemux needs a fix somewhere?

Sure it does, as the options for the NVIDIA HEVC encoder plugin were copied from the H.264 one without any chance to test whether they work or not. If we're lucky, it will be enough to remove non-working options. If we're not, it might be necessary to upgrade or patch the bundled FFmpeg.

According to the log provided, dual-pass definitely doesn't work with Avidemux, the only missing bit of info is whether the stats file is populated or empty.

It would be valuable to know whether some of rate control modes which don't use presets (i.e. "constant quality", "constant bitrate" and "variable bitrate") in single-pass mode work.

Quote from: eumagga0x2a on November 29, 2020, 04:05:50 PM

Quote from: Jiehfeng on November 29, 2020, 04:03:20 PMTried that as well, used the slow preset which says it uses 2 pass, the -rc 2 pass seems to be deprecated.

If there was no stats file, it wasn't a 2-pass mode.

Should that file also appear by the output file? I didn't see any stats file as such.

Quote from: eumagga0x2a on November 29, 2020, 04:25:20 PMAccording to the log provided, dual-pass definitely doesn't work with Avidemux, the only missing bit of info is whether the stats file is populated or empty.

It would be valuable to know whether some of rate control modes which don't use presets (i.e. "constant quality", "constant bitrate" and "variable bitrate") in single-pass mode work.

Tried all three with 2 pass disabled; constant quality at 0 kind of worked, but the bitrate is much lower than expected at about 6000kbps (same 26000kbps source file) and the quality is alright but not great, constant bitrate and variable bitrate with 2 pass disabled gives horrible quality outputs.

Quote from: Jiehfeng on November 30, 2020, 09:59:26 AMShould that file also appear by the output file? I didn't see any stats file as such.

In Avidemux, the stats (and mbtree) file is written to the location of the output. ffmpeg writes its ffmpeg2pass-*.log and .log.mbtree to the location of the input file.

Quote from: Jiehfeng on November 30, 2020, 09:59:26 AMconstant bitrate and variable bitrate with 2 pass disabled gives horrible quality outputs.

Horrible quality no matter which bitrate you specify? The default value of 5000 kbit/s is very low for 4K at 60 fps (it was chosen with 1080p at 25 fps in mind).

Quote from: eumagga0x2a on November 30, 2020, 10:31:16 AM

Quote from: Jiehfeng on November 30, 2020, 09:59:26 AMconstant bitrate and variable bitrate with 2 pass disabled gives horrible quality outputs.

Horrible quality no matter which bitrate you specify? The default value of 5000 kbit/s is very low for 4K at 60 fps (it was chosen with 1080p at 25 fps in mind).

Yeah I specified the same bitrate as the source file, about 26000kbps and the output was horrible with around 600kbps.

@eumagga0x2a
This caught my eye yesterday. I'll reply on nvenc/avidemux/ffmpeg first if it helps.

I've been using nvenc for quite a while through ffmpeg, voukoder plugin (ffmpeg frontend/connector) for Adobe, and rarely, Avidemux. 99% of the time it was the h264 encoder, and a couple of times hevc one. But I always used either CQP quality parameter, or the regular target/max VBR. Never have I tried using it with merely a profile and no any additional rate control.

The only few times I've used NVENC in Avidemux was with constant quality mode too.
Following Jiehfeng's reply, I've tried and I'm getting sort of similar results.

For H264 NVENC, at default setting, Avidemux actually produces a decent high bitrate file. For one high detail and movement 4k60 test file I was getting up to 250Mb/s.

For HEVC NVENC however, without manually specifying rate/quality, it results in a very low bitrate file, 1~4Mb/s.

That caught my interest and I've tried the same with ffmpeg alone. No matter what I tried, for both h264_nvenc and hevc_nvenc, I was getting the same result with ridiculously low bitrate.
I've played with syntax specifying presets and profiles in different ways, but nothing short of providing actual rate or quality value helped.

I don't know if it's the ffmpeg implementation that's at fault or encoder itself. I didn't find much mention of this online, I guess in part because those who would use nvenc through ffmpeg do it in a more or less sophisticated way and would imply some sort of rate control.

The only quote I found was this, from someone:

Quote from: serilain on some superuser replyMy experience with using nvenc is that you need to tell it what bitrate you want -- it defaults to VBR, which is fine, but no amount of tuning makes up for the fact that it always wants to give you a 2M average bitrate no matter what resolution file you feed it. It seems like a flaw in the encoder; it performs predictably in every other respect, but it needs (for example) -b:v 4M for a 720p file or -b:v 8M for 1080p. You could probably lower those a little bit if you want, too.

Maybe it's just their assessment.
Anyway, considering this, it is surprising for me that h264_nvenc works fine with default setting (I mean preset based rate control) in Avidemux.

Regarding hevc_nvenc in Avidemux, as I said it fails to set proper bitrate with presets (I've tried a couple), but works fine with CQ/VBR.
On another note, b-frames are only supported by the encoder in RTX2000 series and onwards.
I've got GTX1080Ti and when turning on b frames, export fails immediately, but that makes sense.

Anyway, this thread lead me to review and improve my nvenc batch scripts as well as some insight.
I think I'll reply to Jiehfeng in a separate reply later as this is becoming a wall of text.

Thank you very much for joining the topic, this helps a lot.

Let's start with the basics, with the CBR mode in HEVC NVENC. Is it broken for you in the same way as for Jiehfeng on Windows?

I've got feedback that the HEVC NVENC encoder plugin in Avidemux works fine on Linux with a GTX 1050Ti, so I suspect that the problem is related either to more recent hardware or to NVIDIA drivers for Windows (or both). You mentioned CQ and VBR which work for you, how about CBR? Can you achieve a good quality by increasing (fixed) bitrate?

Ok, first, I was wrong about VBR not having the problem, I was testing just before replying, and somehow it slipped my attention. Both CBR and VBR have the low bitrate problem for me.
However, I then retested with different types of files, with different results.

After testing with 4 types of files, the possible suspect is variable framerate?

1. nvidia shadowplay game footage, 4k, variable 60fps > low bitrate issue
2. sony camera 1080p, 25 or 50 constant fps > ok
3. 4k game clip exported from Premiere, constant 60fps > ok
4. android phone video, 4k variable 30 fps > low bitrate issue

Thank you, very interesting findings. What I struggle to verify, if the output bitrate in CBR mode is extremely low, does increasing the bitrate in settings of the plugin have no effect whatsoever?

The variable FPS problem may be related to timebase (very low numerator: bad).

In other words: inserting the "Resample FPS" filter with a standard target FPS value should always(?) work around the problem.

(It is perfectly possible that HEVC NVENC test results on Linux were based on CFR source videos only)

Well, it scales but not linearly, I guess. These are Kb/s. It's a short 4k60 VFR clip, not much movement.


original          44071k

target          result
5000k          1676k
15000k          1907k
35000k          2061k
250000k          3738k

Quote from: eumagga0x2a on December 01, 2020, 04:50:47 PMIn other words: inserting the "Resample FPS" filter with a standard target FPS value should always(?) work around the problem.

Yes, just tried and confirmed. Tested on same 4k60 game clip and phone 4k30 clip. When adding the filter, in the value window, the framerate was initially fractional, like 60.12 or 30.02, so obviously I changed it to 60 and 30. I used CBR with target of 50Mb/s. The end files came with approx. 46Mb/s and 51Mb/s bitrates, which is about the same kind of +- variance I would normally get with VBR target (I guess there's also how ffprobe/mediainfo counts it)

Thank you very much, the ball is in my park then, timebase != 1/fps messes up rate control.

@Jiehfeng, @Urik, I've changed (https://github.com/mean00/avidemux2/commit/388aa89adaac1d636f36a32a6657d8f66673d6e3) the way how time base is calculated in the core video encoder code mostly back to the state of affairs prior to [coreVideoEncoderFFmpeg] Use fraction to convert between microseconds and lav ticks, use timebase from FilterInfo instead of deriving it from frame increment to set context, remove redundant timebase-related part of presetContext() (https://github.com/mean00/avidemux2/commit/6c66cfd6c0578a4844c63194ac50d276c61f6a10).

Once fresh nightlies become available, please test and provide feedback whether the change helps with HEVC NVENC encoder and if it does, whether the present-based rate control becomes viable or only the usual CBR, CQ and VBR modes are functional (and whether 2-pass works).

Thanks a lot in advance.

Cool, will do.
Does that mean that h264_nvenc and hevc_nvenc behave in a slightly different way, since you mentioned before the Avidemux code is kind of similar for both? Since it wasn't affecting h264_nvenc.

I've got a few more questions, just of curiosity

1) why is GOP length limited to 250 for both encoders in Avidemux? Is there actual nvenc-related reason, or is it because 250 is one of common/default values for FFMPEG? I haven't found any info on what the max value can be for ffmpeg or nvenc, the max I tried myself was 300 (through ffmpeg)

2) -rc-lookahead, assuming it's undefined, it means the encoder decides itself which value to use (auto)?
3) in VBR mode, how is -buffsize set?

Personally, I've been following very rudimentary rules of having lookahead equal GOP length, max bitrate=target bitrate*1.5 and buffsize=max bitrate*2 just off what I've read online

Oops. I see, dual-pass is currently completely fake both in H264 NVENC and HEVC NVENC encoder plugins (it is not setup as such at all), and when implemented, it doesn't work (the stats file is empty). rc-lookahead needs to be added.

Will try to fix all that ASAP.

A brief status update: the H.264 NVENC part is mostly done, will start with the HEVC one shortly. Most options I add I cannot test on my equipment, so take some fun once the changes land and nightlies get refreshed for granted.

Quote from: Urik on December 03, 2020, 03:10:39 PM1) why is GOP length limited to 250 for both encoders in Avidemux? Is there actual nvenc-related reason, or is it because 250 is one of common/default values for FFMPEG?

No particular reason, maybe just personal preferences for seek granularity. 250 is indeed the default GOP size value FFmpeg sets for *_nvenc if the user doesn't ask for a different one. I've increased the limit to 1000 in my local codetree now.

Quote from: Urik on December 03, 2020, 03:10:39 PM2) -rc-lookahead, assuming it's undefined, it means the encoder decides itself which value to use (auto)?

Presumably lookahead (when supported) is completely disabled then. Unsure, though. BTW, maximum lookahead value supported by NVENC is 31 the number of consecutive B-frames subtracted.

Quote from: Urik on December 03, 2020, 03:10:39 PM3) in VBR mode, how is -buffsize set?

It is set to twice the average bitrate.

Quote from: eumagga0x2a on December 08, 2020, 02:11:47 PMBTW, maximum lookahead value supported by NVENC is 31 the number of consecutive B-frames subtracted.

Interesting. I looked up some official Nvidia docs and indeed they mention it "can be up to 32". Many forum posts online specifically list 32 in ffmpeg syntax. Before I didn't even think to specifically look for nvenc lookahead info, thought it's the same as the one for x264 or whatever. Turns out there's plenty of interesting stuff in official Nvidia documentation I didn't care to read before.

The setup of the both NVENC encoder plugins has been reworked now: [ffNvEnc] Fix numerous issues in encoder configuration, add more options, deduplicate code (https://github.com/mean00/avidemux2/commit/284b2afd7b54a4fc524f8390204742676cd3e4a7), only the H264 part tested so far and only on Linux.

A fresh win64 nightly has been uploaded, please test.

https://avidemux.org/nightly/win64/

I've tried it and I'm a bit confused, is bitrate input field supposed to be available when RC Mode is set to Controlled by Preset? That applies to both HEVC and h264 nvenc options.
So with defaults being 5000 for HEVC and 10000 for h264, the output gets that exact bitrate, kind of like VBR/CBR mode. (I left preset at High Quality at all times but it doesn't seem to make a difference)

Quote from: Urik on December 13, 2020, 12:00:45 PMis bitrate input field supposed to be available when RC Mode is set to Controlled by Preset?

Yes, it must be available. It was my error to disable it, I misunderstood the description of the corresponding option in *_nvenc encoders in libavcodec. Presets control the strategy how the encoder tries to achieve the goal, not the goal itself. The bitrate specifies the desired average bitrate then.

Have you checked whether lookahead works with your hardware? If enabled (and supported), it should result in keyframes being placed not simply at fixed intervals but depending on changes in picture content.

Quote from: eumagga0x2a on December 13, 2020, 11:09:14 PMHave you checked whether lookahead works with your hardware?

I'll check soon, I've been caught up in reinstalling windows last couple days

Quote from: eumagga0x2a on December 13, 2020, 11:09:14 PMIf enabled (and supported), it should result in keyframes being placed not simply at fixed intervals but depending on changes in picture content.

Is that what it really does? I've been using it in ffmpeg and voukoder plugin, and I used to think that it just affects bitrate prediction in VBR/CQP modes.
Then, more recently, I read this in Nvidia's guide for OBS, where lookahead is just an on/off switch

QuoteThis allows the encoder to dynamically select the number of B-Frames, between 0 and the number of B-Frames you specify.

So you're saying it affects scenecut? I thought that's controlled by -no-scenecut (in ffmpeg) alone, or it's different since Avidemux's dealing directly with nvenc?

My graphics card doesn't support lookahead and keyframes are always placed at each keyint-th frame, no matter what the content and how detrimental the choice may be for compression. This means -no-scenecut has no effect as it tries to disable a feature which doesn't exist in the first place. In my understanding lookahead = 0 disables scenecut detection as well, but I may be easily wrong here as I cannot test myself.

Avidemux just wraps libavcodec *_nvenc encoders, it doesn't interact with NVENC directly.

It seems you are right, from what I tested. I used a short test clip with two definitive scene cuts. With lookahead at 0 there were no I-frames placed, then I tried lookahead of 31 and 10, and they were placed exactly where needed.
I tested both h264 and hevc, some were with VBR and some with just high preset + target rate. Maybe I tested CQP too, I don't remember.
I used to think that scene cuts were always on by default, or at least on "better" presets. With voukoder plugin (ffmpeg-based) and my ffmpeg scripts I just always used "slow" preset so I thought that makes the difference, because I had the auto I-frames. I guess this could've been just because I always specified lookahead.

Nvidia's Shadowplay screen recording software has no scenecut I-frames, GOP of 30 (at least for 60fps) and no B-frames. So, presumably they don't use lookahead then. Adobe added nvenc export into their software this year, and it's the same story there.

Just to clarify, when you say libavcodec, it's the same thing that's in FFMPEG, right?

One more thing, in Avidemux, for both encoders, in variable bitrate mode, Quality value entry field is now unlocked, which confused me again. Is it supposed to be this way? It has an effect even in VBR mode?

Quote from: Urik on December 15, 2020, 07:31:05 PMWith lookahead at 0 there were no I-frames placed, then I tried lookahead of 31 and 10, and they were placed exactly where needed.

Thanks for confirmation.

Quote from: Urik on December 15, 2020, 07:31:05 PMI used to think that scene cuts were always on by default, or at least on "better" presets. With voukoder plugin (ffmpeg-based) and my ffmpeg scripts I just always used "slow" preset so I thought that makes the difference, because I had the auto I-frames. I guess this could've been just because I always specified lookahead.

The "slow" preset is "hq" + what NVIDIA calls "2-pass" (which is not 2-pass one would think of). The "slow" preset is definitely slower on my hardware than just "hq" despite no scenecut and no lookahead support, so the encoder is putting more effort in... something.

If you happen to find some spare time, it would be indeed interesting to determine experimentally whether "slow" on your hardware implies scenecut detection even with lookahead option not specified on the ffmpeg command line.

Quote from: Urik on December 15, 2020, 07:31:05 PMJust to clarify, when you say libavcodec, it's the same thing that's in FFMPEG, right?

Yes, the same library. FFmpeg = libavutil + libavcodec + libavformat + libavfilter + libswscale + libpostproc + libavdevice + libavresample + libswresample + a set of command-line frontends to use all of them.

Avidemux bundles libavutil, libavcodec, libavformat, libswscale and libpostproc. Some parts are patched to expose internal, highly sophisticated functions to external usage by Avidemux (very few patches actually modify behaviour).

Quote from: Urik on December 15, 2020, 07:31:05 PMOne more thing, in Avidemux, for both encoders, in variable bitrate mode, Quality value entry field is now unlocked, which confused me again. Is it supposed to be this way? It has an effect even in VBR mode?

This is typical corner cutting. The sematics of the parameter is silently switched from constant quantisation parameter (qp) in "Constant Quantizer" (constqp) mode to constant quality (cq) in "Variable Bitrate" mode. It should have a major effect in VBR mode, with "0" meaning "automatic quality".

Quote from: eumagga0x2aThe "slow" preset is "hq" + what NVIDIA calls "2-pass" (which is not 2-pass one would think of)

Yes, apart from the description, nothing about it resembles 2-pass (in the regular sense) at all. It just goes in one pass in ffmpeg, perhaps just slower.

Quote from: eumagga0x2a on December 16, 2020, 02:36:20 PMIf you happen to find some spare time, it would be indeed interesting to determine experimentally whether "slow" on your hardware implies scenecut detection even with lookahead option not specified on the ffmpeg command line.

I think I tested yesterday, but at this point I start to forget what I tested because of multiple settings etc.

I re-tested, and it seems that in Avidemux there's no scenecut detection regardless of preset.
The rate control mode doesn't seem to affect it, as well as additional b-frame related settings and weighted prediction.

I also tested same with my ffmpeg scripts, where for the test I removed lookahead and did not define gop, and there, in

• h264_nvenc > I DO get scenecut detection in Slow and hq preset, but don't get it with default p4 preset.
• hevc_nvenc > I DON'T get scenecut detection even with slow preset (without lookahead)
  

Settings were the same between h264/hevc apart from omission of b-frames on hevc and main profile instead of high.

My ffmpeg is fairly new though, it's a December "git" build (not "release") and it even requires a fairly new driver (because of API version or something). It has these extra "p" presets

Quote-preset            <int>        E..V...... Set the encoding preset (from 0 to 18) (default p4)
     default         0            E..V......
     slow            1            E..V...... hq 2 passes
     medium          2            E..V...... hq 1 pass
     fast            3            E..V...... hp 1 pass
     hp              4            E..V......
     hq              5            E..V......
     bd              6            E..V......
     ll              7            E..V...... low latency
     llhq            8            E..V...... low latency hq
     llhp            9            E..V...... low latency hp
     lossless        10           E..V......
     losslesshp      11           E..V......
     p1              12           E..V...... fastest (lowest quality)
     p2              13           E..V...... faster (lower quality)
     p3              14           E..V...... fast (low quality)
     p4              15           E..V...... medium (default)
     p5              16           E..V...... slow (good quality)
     p6              17           E..V...... slower (better quality)
     p7              18           E..V...... slowest (best quality)

So I guess it's all kind of tied to driver support, and GPU family / nvenc generation.

I've noticed there's a new VC++ build (of avidemux) now, since personally (normally) I always use VC, I'll be using the updated one now, cool.

Quote from: Urik on December 16, 2020, 05:06:12 PMI re-tested, and it seems that in Avidemux there's no scenecut detection regardless of preset.

Was it with lookahead set to 0?

Quote from: Urik on December 16, 2020, 05:06:12 PMI also tested same with my ffmpeg scripts, where for the test I removed lookahead and did not define gop, and there, in

• h264_nvenc > I DO get scenecut detection in Slow and hq preset, but don't get it with default p4 preset.
• hevc_nvenc > I DON'T get scenecut detection even with slow preset (without lookahead)

Very interesting, thank you for sharing your findings.

Support for the new P1 - P7 presets has been added in FFmpeg only on the trunk so far, Avidemux uses the 4.2.x branch.

Quote from: eumagga0x2a on December 16, 2020, 08:56:16 PMWas it with lookahead set to 0?

Yes. Just tested again. And then with lookahead - even value of 1 is enough to trigger scenecut. I used GOP of 250 in most tests so it's more apparent.

Hey, sorry for the inactivity. Tried the latest nightly build and the Constant Bitrate mode, works perfectly now!

Quote from: Jiehfeng on December 17, 2020, 02:57:21 PMTried the latest nightly build and the Constant Bitrate mode, works perfectly now!

Great, glad to hear that. I hope that other, more useful RC modes work fine too (CBR is obviously the worst one quality- and efficiency-wise, but good for troubleshooting).

Quote from: eumagga0x2a on December 17, 2020, 07:51:08 PM

Quote from: Jiehfeng on December 17, 2020, 02:57:21 PMTried the latest nightly build and the Constant Bitrate mode, works perfectly now!

Great, glad to hear that. I hope that other, more useful RC modes work fine too (CBR is obviously the worst one quality- and efficiency-wise, but good for troubleshooting).

Just tried the others, all work fine!

Thank you for fixing NVidia HEVC settings, it encodes great in Avidemux GUI application now.

Yet there are problems due to some kind of Nvidia H264 and Nvidia HEVC encoders clash.
I've checked with the latest avidemux_2.7.8 r210306_win64.exe and the problem is still there.

• NVidia HEVC video output jobs are wrongly encoded as Nvidia H264.
• NVidia HEVC saved defaults (Ctrl+Alt+D) are loaded as Nvidia H264.

Looks like the problem is that both encoders share the same name ffNvEnc, see log entries below.
[videoEncoder6]Name :ffNvEnc ApiVersion :7 Description :Nvidia hw encoder
[nvCheckDll] 09:39:52-136 checking nvcuda.dll-> 1
[nvCheckDll] 09:39:52-136 checking nvEncodeAPI64.dll-> 1
[VideoEncoder6] Registered filter C:\Program Files\Avidemux 2.7 Qt 64 bits\plugins\videoEncoders\libADM_ve_ffNvencH264.dll as  Nvidia hw encoder
[videoEncoder6]Name :ffNvEnc ApiVersion :7 Description :Nvidia hw encoder
[nvCheckDll] 09:39:52-137 checking nvcuda.dll-> 1
[nvCheckDll] 09:39:52-138 checking nvEncodeAPI64.dll-> 1
[VideoEncoder6] Registered filter C:\Program Files\Avidemux 2.7 Qt 64 bits\plugins\videoEncoders\libADM_ve_ffNvencHEVC.dll as  Nvidia hw encoder
My current workaround is to change the extension of the libADM_ve_ffNvencH264.dll so it is ignored by avidemux but it would be nice to have this sorted out.

Indeed, a pity that no one reported this bug during the last release cycle (my NVIDIA hardware doesn't support encoding to HEVC so that this went unnoticed). If you can handle a hex editor, please replace the byte sequence

66 66 4E 76 45 6E 63 00
with

66 66 4E 76 45 6E 34 00
(ffNvEnc --> ffNvEn4) in libADM_ve_fNvencH264.dll and with

66 66 4E 76 45 6E 35 00
(ffNvEnc --> ffNvEn5) in libADM_ve_ffNvencHEVC.dll respectively as a workaround to resolve the conflict for now.

Should be fixed by this commit (https://github.com/mean00/avidemux2/commit/d7ecad79c55e6c21d6462bc69d2d0491efe3c184). The NVENC encoder name in old project files needs to be edited to make them compatible with future Avidemux builds.

Quote from: eumagga0x2a on March 09, 2021, 08:06:35 PMIndeed, a pity that no one reported this bug during the last release cycle (my NVIDIA hardware doesn't support encoding to HEVC so that this went unnoticed). If you can handle a hex editor, please replace the byte sequence

Code Select Expand

66 66 4E 76 45 6E 63 00
with

Code Select Expand

66 66 4E 76 45 6E 34 00
(ffNvEnc --> ffNvEn4) in libADM_ve_fNvencH264.dll and with

Code Select Expand

66 66 4E 76 45 6E 35 00
(ffNvEnc --> ffNvEn5) in libADM_ve_ffNvencHEVC.dll respectively as a workaround to resolve the conflict for now.

Renaming the first occuranace of ffNvEnc\0 in both files as per instructions above did the trick, thank you.
Encoders are now recognized with their new names and there are no more problems with default settings or jobs (checked using both encoders).