nVidia HEVC encoder not working very well

[Shira]

Hi, been using Avidemux for a long time, mostly encoding game recordings using H.264 nvidia (GTX 660) encoder in avidemux. Today I got a 1070 GTX so I was excited to try the h.265 encoder, but there's something very wrong, I'm testing with a short video file, the encoder is very fast but when I play the video it's very choppy when I try to skip back/forward, the image freezes for 3-4 seconds or more then continues. When using the software based encoder instead, there are no issues with the encoded file.
I tried also with another app: MediaCoder, using also NVENC H.265, no issues there.

Some data:

Avidemux H.265+NVIDIA:

Format                      : Matroska
Format version              : Version 4 / Version 2
File size                   : 34.4 MiB
Duration                    : 1 min 50 s
Overall bit rate            : 2 622 kb/s
Writing application         : Lavf57.25.100
Writing library             : Lavf57.25.100 / Lavf57.25.100

Video
ID                          : 1
Format                      : HEVC
Format/Info                 : High Efficiency Video Coding
Format profile              : Main@L4@High
Codec ID                    : V_MPEGH/ISO/HEVC
Duration                    : 1 min 50 s
Bit rate                    : 2 378 kb/s
Width                       : 640 pixels
Height                      : 360 pixels
Display aspect ratio        : 16:9
Frame rate mode             : Constant
Frame rate                  : 23.976 (24000/1001) FPS
Color space                 : YUV
Chroma subsampling          : 4:2:0
Bit depth                   : 8 bits
Bits/(Pixel*Frame)          : 0.430
Stream size                 : 31.2 MiB (91%)
Default                     : Yes
Forced                      : No
DURATION                    : 00:01:50.193000000

MediaCoder H.265+NVIDIA:

Format                      : Matroska
Format version              : Version 4 / Version 2
File size                   : 14.2 MiB
Duration                    : 1 min 50 s
Overall bit rate            : 1 080 kb/s
Encoded date                : UTC 2016-12-17 04:49:48
Writing application         : mkvmerge v9.1.0 ('Little Earthquakes') 64bit
Writing library             : libebml v1.3.3 + libmatroska v1.4.4

Video
ID                          : 1
Format                      : HEVC
Format/Info                 : High Efficiency Video Coding
Format profile              : Main@L2.1@Main
Codec ID                    : V_MPEGH/ISO/HEVC
Duration                    : 1 min 49 s
Bit rate                    : 1 015 kb/s
Width                       : 640 pixels
Height                      : 360 pixels
Display aspect ratio        : 16:9
Frame rate mode             : Constant
Frame rate                  : 23.976 (24000/1001) FPS
Original frame rate         : 23.000 FPS
Color space                 : YUV
Chroma subsampling          : 4:2:0
Bit depth                   : 8 bits
Bits/(Pixel*Frame)          : 0.184
Stream size                 : 13.3 MiB (94%)
Default                     : Yes
Forced                      : No

Avidemux H.265 cpu:

Format                      : Matroska
Format version              : Version 4 / Version 2
File size                   : 12.2 MiB
Duration                    : 1 min 50 s
Overall bit rate            : 926 kb/s
Writing application         : Lavf57.25.100
Writing library             : Lavf57.25.100 / Lavf57.25.100

Video
ID                          : 1
Format                      : HEVC
Format/Info                 : High Efficiency Video Coding
Format profile              : Main@L2.1@Main
Codec ID                    : V_MPEGH/ISO/HEVC
Duration                    : 1 min 50 s
Bit rate                    : 716 kb/s
Width                       : 640 pixels
Height                      : 360 pixels
Display aspect ratio        : 16:9
Frame rate mode             : Constant
Frame rate                  : 23.976 (24000/1001) FPS
Color space                 : YUV
Chroma subsampling          : 4:2:0
Bit depth                   : 8 bits
Bits/(Pixel*Frame)          : 0.130
Stream size                 : 9.42 MiB (77%)
Writing library             : x265 1.9:[Windows][GCC 4.9.1][64 bit] 8bit
Encoding settings           : wpp / ctu=64 / min-cu-size=8 / max-tu-size=32 / tu-intra-depth=1 / tu-inter-depth=1 / me=3 / subme=5 / merange=16 / no-rect / no-amp / max-merge=2 / temporal-mvp / early-skip / rdpenalty=0 / no-tskip / no-tskip-fast / strong-intra-smoothing / no-lossless / no-cu-lossless / no-constrained-intra / no-fast-intra / open-gop / no-temporal-layers / interlace=0 / keyint=250 / min-keyint=25 / scenecut=40 / rc-lookahead=40 / lookahead-slices=0 / bframes=3 / bframe-bias=0 / b-adapt=1 / ref=3 / limit-refs=3 / no-limit-modes / weightp / weightb / aq-mode=2 / qg-size=32 / aq-strength=1.00 / cbqpoffs=0 / crqpoffs=0 / rd=3 / psy-rd=1.00 / rdoq-level=0 / psy-rdoq=0.00 / signhide / deblock / sao / no-sao-non-deblock / b-pyramid / cutree / no-intra-refresh / rc=crf / crf=20.0 / qcomp=0.60 / qpmin=0 / qpmax=51 / qpstep=4 / ipratio=1.40 / pbratio=1.30
Default                     : Yes
Forced                      : No
DURATION                    : 00:01:50.360000000

[Jan Gruuthuse]

Can you please state your used OS, version and if it is 32- or 64-bit operating?
Upload the sample source video to test with?

ps.: upload:
Use a free dropbox account, https://www.wetransfer.com/, mega or similar web service (free public access, without registration to download your uploaded video) thank you.

[Shira]

Hi, using latest versions

Avidemux 64-bit 2.6.15
Windows 10 x64 Pro 1607

Avidemux H265
https://drive.google.com/open?id=0B0Kd3pQSz9EkaXQ2MHh4ellIMVk

Mediacoder H265
https://drive.google.com/open?id=0B0Kd3pQSz9EkRGlZSjJlUXZDM0U

[eumagga0x2a]

The first video triggers the same error as mentioned in https://github.com/mpv-player/mpv/issues/3440, which implies a muxing issue. The seeking to keyframes is totally broken with this video in Avidemux, it navigates to P-frames instead of intra (but displays correct frame type).

[Shira]

Someone mentioned VLC there, so I tried, it kinda does the same:



and thumbnail preview:



Need me to keep the test files up for a while?

EDIT: same issue with H264 (nvidia enc)

[eumagga0x2a]

Once Mean, the only one whe can understand and fix the issue, has downloaded the samples (maybe this has already happend?), you can take them offline, sure.

[mean]

For some reasons, all frames are tagged as intra

[eumagga0x2a]

For some reasons, all frames are tagged as intra

But UI_setFrameType gets it right, right?

edit: I wonder if it is possible to workaround such cases rechecking the frame type at the target PTS and iterating until there is a true intra.

[mean]

Seems fixed in the win64 nightly

[eumagga0x2a]

What is required to cross-build with Nvenc support in MXE?

[mean]

Just the nvenc header
nvEncodeAPI.h

[eumagga0x2a]

Thank you, just realized how easy it is (the header turns out to be available via rpmfusion.org, just needed the latest nvidia binary drivers to use the feature). Trying to figure out how to build on Linux first (the Debian-specific value for PATHS  ââ,¬â€œ /usr/include/x86_64-linux-gnu ââ,¬â€œ must be replaced by /usr/include/nvenc on Fedora, but then the ffmpeg configure fails to find the header).

[eumagga0x2a]

Quick and very dirty workaround works:

Code Select Expand

# ln -s /usr/include/nvenc/nvEncodeAPI.h /usr/include/

Building now.

[eumagga0x2a]

I've opened a pull request for a (hopefully) proper fix for nvenc detection and ffmpeg configuration.

I'm very impressed by the astonishing speed of encoding in Avidemux with NVENC on Linux (h264 only, HEVC is not available with my hardware). Confirming the issue reported by Shira fixed.

[Shira]

Seems fixed in the win64 nightly

where do you get the nightlies?