invalid h264 extradata buffer passed to ffmpeg

[ajschult]

See https://ffmpeg.org/trac/ffmpeg/ticket/1971
and http://ffmpeg.org/doxygen/trunk/group__lavc__decoding.html#g9207dd7cb4ee4416f8fd1f7a63af34b7

[ajschult]

valgrind output

Code Select Expand


Conditional jump or move depends on uninitialised value(s)
    at 0x51EFF2F: ff_h264_decode_nal (h264.c:207)
    by 0x525491D: decode_nal_units (h264.c:3796)
    by 0x51F24AF: ff_h264_decode_extradata (h264.c:1046)
    by 0x51F2765: ff_h264_decode_init (h264.c:1105)
    by 0x53C2D55: avcodec_open2 (utils.c:1015)
    by 0x53C2120: avcodec_open (utils.c:797)
    by 0x4C32F04: decoderFFH264::decoderFFH264(unsigned int, unsigned int, unsigned int, unsigned int, unsigned char*, unsigned int) (in /usr/lib64/libADM_coreVideoCodec6.so)
    by 0x4C33FC5: ADM_coreCodecGetDecoder(unsigned int, unsigned int, unsigned int, unsigned int, unsigned char*, unsigned int) (in /usr/lib64/libADM_coreVideoCodec6.so)
    by 0x452361: ADM_EditorSegment::addReferenceVideo(_VIDEOS*) (in /usr/bin/avidemux3_qt4)
    by 0x44B91B: ADM_Composer::addFile(char const*) (in /usr/bin/avidemux3_qt4)
    by 0x43605E: A_openAvi(char const*) (in /usr/bin/avidemux3_qt4)
    by 0x461BD7: automation() (in /usr/bin/avidemux3_qt4)

Conditional jump or move depends on uninitialised value(s)
    at 0x51EFF72: ff_h264_decode_nal (h264.c:211)
    by 0x525491D: decode_nal_units (h264.c:3796)
    by 0x51F24AF: ff_h264_decode_extradata (h264.c:1046)
    by 0x51F2765: ff_h264_decode_init (h264.c:1105)
    by 0x53C2D55: avcodec_open2 (utils.c:1015)
    by 0x53C2120: avcodec_open (utils.c:797)
    by 0x4C32F04: decoderFFH264::decoderFFH264(unsigned int, unsigned int, unsigned int, unsigned int, unsigned char*, unsigned int) (in /usr/lib64/libADM_coreVideoCodec6.so)
    by 0x4C33FC5: ADM_coreCodecGetDecoder(unsigned int, unsigned int, unsigned int, unsigned int, unsigned char*, unsigned int) (in /usr/lib64/libADM_coreVideoCodec6.so)
    by 0x452361: ADM_EditorSegment::addReferenceVideo(_VIDEOS*) (in /usr/bin/avidemux3_qt4)
    by 0x44B91B: ADM_Composer::addFile(char const*) (in /usr/bin/avidemux3_qt4)
    by 0x43605E: A_openAvi(char const*) (in /usr/bin/avidemux3_qt4)
    by 0x461BD7: automation() (in /usr/bin/avidemux3_qt4)


and in the console, I see

Code Select Expand


  [extraHeader]  [FLV] found some extradata 39

0000 : .M@.��..gM@.� 4.  01 4d 40 15 ff e1 00 18 67 4d 40 15 bb 20 34 16
0010 : �.��...@..�x�r@  f5 11 80 88 00 00 1f 40 00 04 e3 f4 78 b1 72 40
0020 : ...h�2�  01 00 04 68 ee 32 c8


which seems to correspond to the extradata in question.  ff_h264_decode_nal gets the last 4 bytes.

[mean]

ah yes, we do the padding for data but not extradata
Will fix that this weekend