"copy" mode: Removing frames from a video results in missing reference frames

[karl]

When I mark a range in the middle of some input file, press "delete" to remove that range, and then save the result using "copy" mode, the resulting video contains damaged frames right before (see (2) below) and/or after (see (3) below) the cut.

I do understand that avidemux can cut H.264 videos only in whole GOPs in "copy" mode, and that is fine with me. The raw material I am cutting consists of GOPs that all contain one Intra frame followed by two B-frames: I-B-B-I-B-B-I-B-B-I-....I-B-B-I - you get the idea.

I noticed several odd behaviours of avidemux:

(1) In the status line at the bottom of the avidemux window, avidemux reports only the very first Intra frame as "Frame type: I-FRM (00)", while all following Intra frames are reported as being "P-FRM (00)". (Other software does not report such, see e.g. http://www.personal-view.com/talks/uploads/FileUpload/1d/a6d44819b02cb42e444f11e00e46b4.jpg for a report on this kind of input file.)

(2) When you mark a range and save, avidemux does check whether the first frame after the cut is an Intra frame, and if it is not, it warns you that this will result in a broken output. That is good. BUT: If the last frame before the cut is a B-frame, avidemux does NOT warn you, even though decoding the B-frame will require the following I frame (or P frame, if there was one)! As a result, the B-frames before the cut will appear broken in the output video.
Simple solution: Display the same warning if the last frame before the cut is a B-frame. The user can just cut at the next P or I frame to do the right thing[TM].

(3) If the user takes care to cut such that the last frame before the cut is an I-frame, and the first frame after the cut is also an I-frame, there are still damaged frames displayed right after the cut, and the reason for this may be that these I-frames are not IDR-frames (see also: http://documentation.apple.com/en/compressor/usermanual/index.html#chapter=13%26section=2%26tasks=true or http://books.google.de/books?hl=de&id=TbtNkb41NYYC&q=IDR#v=snippet&q=IDR&f=false page 114ff.).
Could it be that avidemux allows to cut on boundaries of open GOPs within a closed GOP, such that frames right after the cut will reference frames that are not existent in the output file?
Possible solution: Identify IDR-frames (look for NAL_IDR_SLICE in ffmpeg), and only allow remaining segments for output to begin at these IDR frames.

[karl]

I looked up how ffmpeg identifies IDR frames. in libavcodec/h264.c there is a function ff_h264_decode_nal() which assigns "h->nal_unit_type= src[0]&0x1F;" from the first byte of the  "src" data pointer it is passed. Elsewhere in ffmpeg, H264Context::nal_unit_type is compared with the constant "NAL_IDR_SLICE" (==5, defined in  libavcodec/h264.h) to differentiate IDR slices from other types.

I don't understand yet how this corresponds to what is done in ADM_coreUtils/src/ADM_infoExtractorH264.cpp getNalType(), where *flags=AVI_KEY_FRAME is set based on a "sliceType". 

[mean]

Some sources (HD TV broadcast for example) never send IDR frames
Instead they send I frames with a recovery period of 0, which makes them IDR

So the function you mention is looking for frame type (I,P,B) and mark I-Slice as P or I depending if recovery time=0 or not, in other words
IDR or not

[karl]

Some sources (HD TV broadcast for example) never send IDR frames

That I found to be true also for the camera output files I intend to cut.

Instead they send I frames with a recovery period of 0, which makes them IDR

Well, so far I've found some inconclusive texts on that, e.g. the author of http://mailman.videolan.org/pipermail/vlc-commits/2010-October/003615.html seems to be convinced that this is just an "approximate" method to substitute for IDRs.

So the function you mention is looking for frame type (I,P,B) and mark I-Slice as P or I depending if recovery time=0 or not, in other words
IDR or not

This explains why avidemux reports the I-frames in my files as being P-frames. Based on this, the awkward conclusion would be that there is no guaranteed-to-work way to cut away any frames from those files without a residual risk of getting defect frames 

Yet, in practice, it seems that cutting away frames has adverse ripple effects only on a few following frames - so I wonder whether there is a "safe margin" that I could choose to make sure the relevant frames are correctly retained.

(Since I do the cutting in the compressed domain only to save storage space, it would not bother me to keep some extra frames before the relevant ones. Rendering of output I do later with encoding, so that's the only point at which I require frame accurateness.)

[mean]

The link you mention is talking about recovery time in general
The function in avidemux checks if recovery time=0

i.e. the former = Decode N frames and drop them and you are good
the latter Decode and dont drop any frame=IDR by definition

The recovery stuff is made for seeking, it does not work for editing i.e. since there is no seek; the player doesnt know it should engage recovery

[karl]

After having done several more experiments, I now think I was probably on the wrong track when assuming that references to frames outside the GOPs I wanted to retain are causing the "defect frames" to show up the in output files.

First, I simplified my test scenario by not cutting away frames from the middle of the file, but instead just marking a region in the middle of a file and saving that into the output. I further simplified my test case by reducing the amount of frames I saved into the output to very few - 4 frames and finally even only one frame.

I found that the I-Frame I set the A-Mark to (which, for the reasons you explained earlier, is displayed to be a "P-Frame" by avidemux) is indeed stored in the output file, but something from the output is displayed even before that first frame, and those "phantom frames before the first wanted frame" look defect. I also noticed that the first wanted frame is scheduled for display not at "0:00:00", but at 0.125 seconds, which 3 * 1s / 24 later than expected.

My new hypothesis is that maybe, those defect frames displayed are residues from the B-frames that precede the I-frame that set "Mark A" at.
Maybe those B-frames even follow the data for the first wanted I-frame in the input file, as B-frames require the following reference frame to be decoded before they can be decoded (this is currently only an idea of mine, I have no evidence to prove that).
Could this be it? Is there an easy way to verify or falsify this hypothesis? Would it be helpful if I attached some minimalistic (e.g. 9 frames long) transport stream file?


I finally also did the same export test marking a "range" of only one frame, but the output of this looked weird, it did not include a single correct decodeable frame. I guess this could be a result of hitting many corner cases, though.

[jfx]

I also noticed this problem. Is there no way to suppress display of several referenced frames before a cut point? This would allow the encoder to write all required frames to the output stream.

The following message contains more information about this problem:

http://lists.mpegif.org/pipermail/mp4-tech/2009-March/008712.html

If the encoder wants the decoder to be aware that some pictures it is using as references are missing, it can use reference picture list modification syntax to explicitly place specific pictures into the list(s) in the correct order.  Doing so will cause the decoder to notice any pictures placed into the lists that have frame_num values that the decoder does not have in its DPB, which will cause the decoder to become aware of which pictures that are being referenced are not available.

[zakk]

I have had problems too, making clean cuts for a few days, but I find the topic hard to follow. Could one of you please write a quick howto in the forum? Where to (or not to) cut and at which type of frames etc... Thanks.

[karl]

@zakk: For me the following work-around works so far:

I avoid cutting pieces from the middle of input video files and saving the remainder.

Instead, I only mark the scenes I want to retain, and export each them into a separate .mkv file - one by one. (This works for me only when using .mkv output, when I use the MP4v2 output, I see defect frames and defect timing in the output.)

Of course, it would be great if such workaround would not be required. 

[zakk]

Thanks. Do you put A on I-frames and B on I or P-frames ? How do you recognize them when Avidemux shows only P ?

[karl]

I put the marker on those frames that avidemux26 jumps to when pressing "cursor up" or "cursor down" - but while this generally works, I still have to skip some frames at the beginning and the end of the resulting output file when using them with "kdenlive", as kdenlive will otherwise jump some frames back and forth at the beginning/end.

[nibbles]

@zakk I don't get that issue anymore where it only shows P-frames.  I get plenty of I-Frames.  Maybe your source file has something odd about it.  You can post a sample if it's not copyrighted.

[Jan Gruuthuse]

I can confirm what zakk is reporting. This seems only to happen in HD TV MPEG-TS streams. Did not see this in SD TV MPEG-TS stream, yet. Can't tell on other formats, don't use these.
Each Satellite TV Recording is on average 20 MB in download
There is one I could find that is showing partial I-FRM / P-FRM
3sat HD
These are not showing I-FRM:
arte HD
bbc HD
ndr HD
phoenix HD
swr HD

[mean]

Pseudo IDR=I with recovery=0
As far as navigating is concerned, avidemux uses pseudo IDR as IDR, hence you can use the scrollbar
To display the frame type, avidemux uses the decoder output and the decoder says it is an I (not IDR) which is considered as a P as far as avidemux is concerned

The only exception is if there is no IDR at all (pseudo or real) , avidemux will handle I as IDR and it will lead to missing ref frames
That can happen for AVCHD files

[zakk]

I did compare with r7745. When hitting ">>" (or "UP" arrow) recent builds do navigate correctly between what r7745 calls I-Frames, and skips P-Frames. It's just harder to know if you are on a "real" P or not.