Nvidia HEVC problems

[eumagga0x2a]

I chose CUDA acceleration when I first started the project, but when exporting, selecting HEVC has it locked to "Software Encoding" and I can't change it. I think the GPU is utilised but I guess it's still Software Encoding according to you.

Looks like Premiere is able to detect that something doesn't work right with the NVIDIA HW accelerated HEVC encoder.

Ah I see, by quality not respected, is it a bad enough difference to not choose HEVC?

As the NVENC is apparently broken with Avidemux (with libavcodec as used in Avidemux) on Windows with the NVIDIA graphics card and drivers you use (and maybe with all driver versions available for this card), it is up to you to either select the only software HEVC encoder available (x265, superior by far to NVENC but very expensive computationally) or to choose a different codec alltogether. By choosing H.264, you pay the price by needing a higher bitrate for given quality (but get a better compatibility in return).

What about Handbrake? Though the GPU load was low at 4%, it wasn't at 0% which is what it was at when the rendering finished. Does it also use Software Encoding for HEVC? The output was fine.

Can't tell, probably Handbrake is using GPU for decoding only. Encoding a video with bare command-line ffmpeg and hevc_nvenc might provide some clues:

Code Select Expand

ffmpeg -i "DriveLetter:\Path\To\SourceVideo" -c:v hevc_nvenc -preset:v hq -c:a copy "DriveLetter:\Path\To\OutputVideo.mkv"

[Jiehfeng]

I chose CUDA acceleration when I first started the project, but when exporting, selecting HEVC has it locked to "Software Encoding" and I can't change it. I think the GPU is utilised but I guess it's still Software Encoding according to you.

Looks like Premiere is able to detect that something doesn't work right with the NVIDIA HW accelerated HEVC encoder.

Ah I see, by quality not respected, is it a bad enough difference to not choose HEVC?

As the NVENC is apparently broken with Avidemux (with libavcodec as used in Avidemux) on Windows with the NVIDIA graphics card and drivers you use (and maybe with all driver versions available for this card), it is up to you to either select the only software HEVC encoder available (x265, superior by far to NVENC but very expensive computationally) or to choose a different codec alltogether. By choosing H.264, you pay the price by needing a higher bitrate for given quality (but get a better compatibility in return).

What about Handbrake? Though the GPU load was low at 4%, it wasn't at 0% which is what it was at when the rendering finished. Does it also use Software Encoding for HEVC? The output was fine.

Can't tell, probably Handbrake is using GPU for decoding only. Encoding a video with bare command-line ffmpeg and hevc_nvenc might provide some clues:

Code Select Expand

ffmpeg -i "DriveLetter:\Path\To\SourceVideo" -c:v hevc_nvenc -preset:v hq -c:a copy "DriveLetter:\Path\To\OutputVideo.mkv"

I see, thanks for the recommendation.

Also, I think I've been mistaken to look at GPU Load, cause in these new GPUs the encoding work is done in a separate area, so I opened up task manager while I was using ffmpeg as you said and the Video Encode section was at 100% the whole time.

The output has a higher bitrate than avidemux, the quality is decent, but not very clear, however not terrible like how it was with avidemux.



Here's what the CMD looked like:

Code Select Expand

C:\Users\alain\Documents\ShareX\Tools>ffmpeg -i "E:\SharePlay\The Witcher 3\The Witcher 3 2020.11.02 - 13.58.08.04.DVR.mp4" -c:v hevc_nvenc -preset:v hq -c:a copy "C:\Users\alain\Desktop\Tester\Witcher.mp4"
ffmpeg version git-2020-05-27-8b5ffae Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 9.3.1 (GCC) 20200523
  configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libdav1d --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libsrt --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --disable-w32threads --enable-libmfx --enable-ffnvcodec --enable-cuda-llvm --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth --enable-libopenmpt --enable-amf
  libavutil      56. 49.100 / 56. 49.100
  libavcodec     58. 87.101 / 58. 87.101
  libavformat    58. 43.100 / 58. 43.100
  libavdevice    58.  9.103 / 58.  9.103
  libavfilter     7. 83.100 /  7. 83.100
  libswscale      5.  6.101 /  5.  6.101
  libswresample   3.  6.100 /  3.  6.100
  libpostproc    55.  6.100 / 55.  6.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'E:\SharePlay\The Witcher 3\The Witcher 3 2020.11.02 - 13.58.08.04.DVR.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: isommp42
    creation_time   : 2020-11-02T08:28:11.000000Z
    date            : 2020
  Duration: 00:07:19.27, start: 0.000000, bitrate: 26934 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, smpte170m/smpte170m/bt470m), 2560x1440 [SAR 1:1 DAR 16:9], 26731 kb/s, 60 fps, 60 tbr, 90k tbn, 120 tbc (default)
    Metadata:
      creation_time   : 2020-11-02T08:28:11.000000Z
      handler_name    : VideoHandle
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 196 kb/s (default)
    Metadata:
      creation_time   : 2020-11-02T08:28:11.000000Z
      handler_name    : SoundHandle
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> hevc (hevc_nvenc))
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
Output #0, mp4, to 'C:\Users\alain\Desktop\Tester\Witcher.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: isommp42
    date            : 2020
    encoder         : Lavf58.43.100
    Stream #0:0(und): Video: hevc (hevc_nvenc) (Main) (hev1 / 0x31766568), yuv420p, 2560x1440 [SAR 1:1 DAR 16:9], q=-1--1, 2000 kb/s, 60 fps, 15360 tbn, 60 tbc (default)
    Metadata:
      creation_time   : 2020-11-02T08:28:11.000000Z
      handler_name    : VideoHandle
      encoder         : Lavc58.87.101 hevc_nvenc
    Side data:
      cpb: bitrate max/min/avg: 0/0/2000000 buffer size: 4000000 vbv_delay: N/A
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 196 kb/s (default)
    Metadata:
      creation_time   : 2020-11-02T08:28:11.000000Z
      handler_name    : SoundHandle
frame=26356 fps= 84 q=31.0 Lsize=   75437kB time=00:07:19.25 bitrate=1406.9kbits/s speed= 1.4x
video:64413kB audio:10537kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.650333%

C:\Users\alain\Documents\ShareX\Tools>

[eumagga0x2a]

Thank you, this means that NVENC actually works for HEVC, but not the usual rate control settings (the output bitrate is very low and quantizer is quite high for a "high quality" preset).

Could you please continue testing with ffmpeg which modes produce usable results and which don't? Does it help to specify a fixed, high enough bitrate?

I also suspect that dual-pass mode is broken (is the stats file really populated or empty?).

Code Select Expand

ffmpeg -h encoder=hevc_nvenc
lists all available options.

PS: Would you please quote only the parts of the text you directly refer to, instead of a complete reply? Thanks!

[Jiehfeng]

Thank you, this means that NVENC actually works for HEVC, but not the usual rate control settings (the output bitrate is very low and quantizer is quite high for a "high quality" preset).

I see, great!

Could you please continue testing with ffmpeg which modes produce usable results and which don't? Does it help to specify a fixed, high enough bitrate?

I tried the lossless option, bitrate is much higher at 59000kbps, looks more or less the same as the source.

Then I tried Constant Bitrate Mode (the regular one, not the hq one) with the same bitrate as the source, the resulting bitrate is lower (as you said that is expected), but the quality is great, almost like the original:



cbr hq also gave similar results, but slightly lower bitrate at 11000kbps.

vbr hq also gave more or less the same result and bitrate.

I also suspect that dual-pass mode is broken (is the stats file really populated or empty?).

Tried that as well, used the slow preset which says it uses 2 pass, the -rc 2 pass seems to be deprecated. I specified the bitrate to the same as the source here as well. Bitrate was around 11000kbps and quality is more or less the same as the source.

So all in all, all the modes I tried gave similar results and all of them look the same to me and to the source. So does this mean avidemux needs a fix somewhere?

[eumagga0x2a]

Tried that as well, used the slow preset which says it uses 2 pass, the -rc 2 pass seems to be deprecated.

If there was no stats file, it wasn't a 2-pass mode.

[eumagga0x2a]

So does this mean avidemux needs a fix somewhere?

Sure it does, as the options for the NVIDIA HEVC encoder plugin were copied from the H.264 one without any chance to test whether they work or not. If we're lucky, it will be enough to remove non-working options. If we're not, it might be necessary to upgrade or patch the bundled FFmpeg.

[eumagga0x2a]

According to the log provided, dual-pass definitely doesn't work with Avidemux, the only missing bit of info is whether the stats file is populated or empty.

It would be valuable to know whether some of rate control modes which don't use presets (i.e. "constant quality", "constant bitrate" and "variable bitrate") in single-pass mode work.

[Jiehfeng]

Tried that as well, used the slow preset which says it uses 2 pass, the -rc 2 pass seems to be deprecated.

If there was no stats file, it wasn't a 2-pass mode.

Should that file also appear by the output file? I didn't see any stats file as such.

According to the log provided, dual-pass definitely doesn't work with Avidemux, the only missing bit of info is whether the stats file is populated or empty.

It would be valuable to know whether some of rate control modes which don't use presets (i.e. "constant quality", "constant bitrate" and "variable bitrate") in single-pass mode work.

Tried all three with 2 pass disabled; constant quality at 0 kind of worked, but the bitrate is much lower than expected at about 6000kbps (same 26000kbps source file) and the quality is alright but not great, constant bitrate and variable bitrate with 2 pass disabled gives horrible quality outputs.

[eumagga0x2a]

Should that file also appear by the output file? I didn't see any stats file as such.

In Avidemux, the stats (and mbtree) file is written to the location of the output. ffmpeg writes its ffmpeg2pass-*.log and .log.mbtree to the location of the input file.

constant bitrate and variable bitrate with 2 pass disabled gives horrible quality outputs.

Horrible quality no matter which bitrate you specify? The default value of 5000 kbit/s is very low for 4K at 60 fps (it was chosen with 1080p at 25 fps in mind).

[Jiehfeng]

constant bitrate and variable bitrate with 2 pass disabled gives horrible quality outputs.

Horrible quality no matter which bitrate you specify? The default value of 5000 kbit/s is very low for 4K at 60 fps (it was chosen with 1080p at 25 fps in mind).

Yeah I specified the same bitrate as the source file, about 26000kbps and the output was horrible with around 600kbps.

[Urik]

@eumagga0x2a
This caught my eye yesterday. I'll reply on nvenc/avidemux/ffmpeg first if it helps.

I've been using nvenc for quite a while through ffmpeg, voukoder plugin (ffmpeg frontend/connector) for Adobe, and rarely, Avidemux. 99% of the time it was the h264 encoder, and a couple of times hevc one. But I always used either CQP quality parameter, or the regular target/max VBR. Never have I tried using it with merely a profile and no any additional rate control.

The only few times I've used NVENC in Avidemux was with constant quality mode too.
Following Jiehfeng's reply, I've tried and I'm getting sort of similar results.

For H264 NVENC, at default setting, Avidemux actually produces a decent high bitrate file. For one high detail and movement 4k60 test file I was getting up to 250Mb/s.

For HEVC NVENC however, without manually specifying rate/quality, it results in a very low bitrate file, 1~4Mb/s.

That caught my interest and I've tried the same with ffmpeg alone. No matter what I tried, for both h264_nvenc and hevc_nvenc, I was getting the same result with ridiculously low bitrate.
I've played with syntax specifying presets and profiles in different ways, but nothing short of providing actual rate or quality value helped.

I don't know if it's the ffmpeg implementation that's at fault or encoder itself. I didn't find much mention of this online, I guess in part because those who would use nvenc through ffmpeg do it in a more or less sophisticated way and would imply some sort of rate control.

The only quote I found was this, from someone:

My experience with using nvenc is that you need to tell it what bitrate you want -- it defaults to VBR, which is fine, but no amount of tuning makes up for the fact that it always wants to give you a 2M average bitrate no matter what resolution file you feed it. It seems like a flaw in the encoder; it performs predictably in every other respect, but it needs (for example) -b:v 4M for a 720p file or -b:v 8M for 1080p. You could probably lower those a little bit if you want, too.

Maybe it's just their assessment.
Anyway, considering this, it is surprising for me that h264_nvenc works fine with default setting (I mean preset based rate control) in Avidemux.

Regarding hevc_nvenc in Avidemux, as I said it fails to set proper bitrate with presets (I've tried a couple), but works fine with CQ/VBR.
On another note, b-frames are only supported by the encoder in RTX2000 series and onwards.
I've got GTX1080Ti and when turning on b frames, export fails immediately, but that makes sense.

Anyway, this thread lead me to review and improve my nvenc batch scripts as well as some insight.
I think I'll reply to Jiehfeng in a separate reply later as this is becoming a wall of text.

[eumagga0x2a]

Thank you very much for joining the topic, this helps a lot.

Let's start with the basics, with the CBR mode in HEVC NVENC. Is it broken for you in the same way as for Jiehfeng on Windows?

I've got feedback that the HEVC NVENC encoder plugin in Avidemux works fine on Linux with a GTX 1050Ti, so I suspect that the problem is related either to more recent hardware or to NVIDIA drivers for Windows (or both). You mentioned CQ and VBR which work for you, how about CBR? Can you achieve a good quality by increasing (fixed) bitrate?

[Urik]

Ok, first, I was wrong about VBR not having the problem, I was testing just before replying, and somehow it slipped my attention. Both CBR and VBR have the low bitrate problem for me.
However, I then retested with different types of files, with different results.

After testing with 4 types of files, the possible suspect is variable framerate?

1. nvidia shadowplay game footage, 4k, variable 60fps > low bitrate issue
2. sony camera 1080p, 25 or 50 constant fps > ok
3. 4k game clip exported from Premiere, constant 60fps > ok
4. android phone video, 4k variable 30 fps > low bitrate issue

[eumagga0x2a]

Thank you, very interesting findings. What I struggle to verify, if the output bitrate in CBR mode is extremely low, does increasing the bitrate in settings of the plugin have no effect whatsoever?

The variable FPS problem may be related to timebase (very low numerator: bad).

[eumagga0x2a]

In other words: inserting the "Resample FPS" filter with a standard target FPS value should always(?) work around the problem.

(It is perfectly possible that HEVC NVENC test results on Linux were based on CFR source videos only)