crash after VDPAU fails to initialize

[ajschult]

Fedora shipped some VDPAU drivers and after installing them, avidemux crashes.  In the console, I get

Code Select Expand


[ADM_getDecoder]  Searching decoder in vdpau (544 x 408, extradataSize:42)...
[ADM_getDecoder]  This is vdpau compatible
[lavc] Build: 3562596
[lavc] Decoder init: h264_vdpau video decoder initialized! (H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 (VDPAU acceleration))
[decoderFFVDPAU]  [VDPAU] Decoder created
[decoderCreate]  ADM_coreVdpau::funcs.decoderCreate(ADM_coreVdpau::vdpDevice,profile,width,height,max_references,decoder) call failed with error=An invalid/unsupported VdpDecoderProfile value was supplied.
[decoderFFVDPAU]  Cannot create VDPAU decoder
[~decoderFFVDPAU]  [VDPAU] Cleaning up
[~decoderFFVDPAU]  [VDPAU] Destroying decoder
avidemux3_qt4: htab.c:83: vlGetDataHTAB: Assertion `handle' failed.


the stack looks like

Code Select Expand


#4  0x00007fffe0976463 in vlGetDataHTAB (handle=handle@entry=0) at htab.c:83
#5  0x00007fffe097825d in vlVdpDecoderDestroy (decoder=0) at decode.c:134
#6  0x00007ffff79cb27a in admVdpau::decoderDestroy (decoder=0)
    at /build/andrew/avidemux/avidemux_2.6.0/avidemux_core/ADM_coreVideoCodec/ADM_hwAccel/ADM_coreVdpau/src/ADM_coreVdpau.cpp:211
#7  0x000000000045eff8 in decoderFFVDPAU::~decoderFFVDPAU() ()
#8  0x000000000045f079 in decoderFFVDPAU::~decoderFFVDPAU() ()
#9  0x0000000000461af1 in ADM_getDecoder(unsigned int, unsigned int, unsigned int, unsigned int, unsigned char*, unsigned int) ()


The decoderFFVDPAU tries to invoke admVdpau::decoderDestroy(VDPAU->vdpDecoder), but vdpDecoder is already NULL (apparently because VDPAU decoder could not actually be created).

Code Select Expand


Index: ADM_ffmpeg_vdpau.cpp
===================================================================
--- ADM_ffmpeg_vdpau.cpp        (revision 8980)
+++ ADM_ffmpeg_vdpau.cpp        (working copy)
@@ -385,11 +385,13 @@
             }
             VDPAU->freeQueue.clear();
         }
-         ADM_info("[VDPAU] Destroying decoder\n");
-         if(VDP_STATUS_OK!=admVdpau::decoderDestroy(VDPAU->vdpDecoder))
+        if (VDPAU->vdpDecoder) {
+            ADM_info("[VDPAU] Destroying decoder\n");
+            if(VDP_STATUS_OK!=admVdpau::decoderDestroy(VDPAU->vdpDecoder))
                 ADM_error("Error destroying VDPAU decoder\n");
-         delete VDPAU;
-         vdpau=NULL;
+        }
+        delete VDPAU;
+        vdpau=NULL;
}
/**
     \fn uncompress


Of course, I'd love for VDPAU to work, but in the absence of that, I'd prefer avidemux to not crash.   

[mean]

Thanks, applied
Is avidemux the only vdpau aware app that has problem ?

[ajschult]

Yes, mplayer seems to be use vdpau happily.