Avidemux Forum

Avidemux => Windows => Topic started by: fish on January 09, 2020, 05:21:40 PM

Title: Can I save this edited file some way in v2.7.5?
Post by: fish on January 09, 2020, 05:21:40 PM
I have just started using ADM 2.7.5 early this week and encountered, 'The saved video is incomplete' because of invalid timestamps. This was while editing a DVB HD file which I know give problems (not only in  ADM I have found). This is a real problem, because instead of getting a flawed file, which in the past I have compensated for by leaving one I-Frame handles at each edit, I now get next to no file saved. I tried opening the saved project script in v2.7.3 but it won't open in v2.7.3.  So a file that should save as 14 minutes, possibly with artifacts for which I have compensated, now saves as a 2 second file, with no option of saving it.
I can't see how this is a good development, i.e. saving a full edit (which may have problems) v saving next to nothing, with no option to save the file despite possible problems. The result of this change is,  I can now only have a 2 second file or nothing. It would be much better to have a option of saving the 14 minute file, with flaws, than being forced to make do with a 2 second file.
Title: Re: Can I save this edited file some way in v2.7.5?
Post by: eumagga0x2a on January 09, 2020, 05:35:40 PM
Do you refer to the latest nightly? Please provide a sample.

If you save to MP4, try setting clock frequency to 90kHz.
Title: Re: Can I save this edited file some way in v2.7.5?
Post by: fish on January 09, 2020, 05:55:54 PM
It might be the latest nightly it's 191222_...4.2.1-vc++. I can redo the edits using v2.7.3 but I just thought I would comment as it seems like a backward step to get next to no file saved, as opposed to a full but flawed file.
Title: Re: Can I save this edited file some way in v2.7.5?
Post by: eumagga0x2a on January 09, 2020, 06:04:08 PM
Please provide information which might help to fix the regression (if 2.7.3 works where the nightly fails).
Title: Re: Can I save this edited file some way in v2.7.5?
Post by: eumagga0x2a on January 09, 2020, 06:08:22 PM
Builds from 2019-12-19 on contain massive changes to the way Avidemux handles fps and timebase, fixing issues like perfectly regular constant fps mp4 files becoming variable frame rate mp4. The changes affect all ffmpeg-based muxers, i.e. mp4, mkv, mpegts, mpegps and webm. The latest nightly from 2020-01-01 is mingw-only, but the changes there affect mostly the flv demuxer.
Title: Re: Can I save this edited file some way in v2.7.5?
Post by: eumagga0x2a on January 09, 2020, 06:32:31 PM
It looks like I haven't committed and pushed updates to the MpegTS, MpegPS and MKV to the repository, they exist only in my local tree. This might be responsible for the failure. Please specify which format you were saving to.
Title: Re: Can I save this edited file some way in v2.7.5?
Post by: fish on January 09, 2020, 06:32:49 PM
I have noticed that v2.7.5, when editing DVB HD files, which have always given problems, such as artifacts and freezes around the edit points, now refuse to save but instead give a warning. It seems to me that saving the full edited video, with problems, would be preferable to saving just the section up to the first problematic edit.
My work round was to save the file in four parts cutting round the problematic frames and then saving that using append.
Title: Re: Can I save this edited file some way in v2.7.5?
Post by: fish on January 09, 2020, 06:34:58 PM
I was saving to .mp4  and the files were originally .ts. No worries, you guys to a fantastic job.
Title: Re: Can I save this edited file some way in v2.7.5?
Post by: eumagga0x2a on January 09, 2020, 06:37:07 PM
Okay, please set clock frequency in the MP4 muxer settings to 90kHz then.

edit: as the input is TS, this means that the problem is with the MpegTS demuxer. There are some uncommitted changes as well, will check... In any case, please delete *.idx2 files to force re-indexing.
Title: Re: Can I save this edited file some way in v2.7.5?
Post by: fish on January 09, 2020, 06:42:00 PM
I'll give that a try in my next edit.
Title: Re: Can I save this edited file some way in v2.7.5?
Post by: eumagga0x2a on January 09, 2020, 06:44:01 PM
Do you still have the source .ts from the last video editing session, which would allow to reproduce the issue?
Title: Re: Can I save this edited file some way in v2.7.5?
Post by: fish on January 09, 2020, 06:53:49 PM
It's already deleted but any DVB HD broadcast file should be good for testing.
I remember a post by someone a year or two ago and the poster had a way of analysing the GOP structure of DVB HD files before and after editing. It showed the frame structure around the edit was getting damaged quite often. ADM isn't the only editor that has trouble reading these files, quite a few won't look at them and others get frames mixed up.
Title: Re: Can I save this edited file some way in v2.7.5?
Post by: eumagga0x2a on January 09, 2020, 07:00:51 PM
That is a pity as not any DVB HD broadcast will do.

Avidemux knows since last summer which B-frames in H.264 streams it may drop and which not, thus no damaged GOPs should occur. The issue you encountered this time is however a completely different one. The timebase provided to the libavformat-based muxer doesn't match the real timestamps.
Title: Re: Can I save this edited file some way in v2.7.5?
Post by: eumagga0x2a on January 09, 2020, 10:32:09 PM
No problems saving a ~1h long freshly recorded DVB HD transport stream with quite a few transmission errors to mp4 so far, please provide a sample once you get hold of a suitable one.
Title: Re: Can I save this edited file some way in v2.7.5?
Post by: fish on January 12, 2020, 01:13:58 AM
That's very strange but from what you describe you don't appear to have removed any sections in the video. The problem does not arise if you simply take a whole DVB-T  HD file and save it in a mp4 container, the problem arises around edits. Try taking 10 mins of the video and remove 5 - 10, 5 second sections.
This has been a long running problem caused by this type of file, see the post 'Broken frames due to H.264 Open-GOP (DVB MPEG-TS)' which describes it very well. What the poster describes concurs exactly with what I see. The problem is with this type of file, I haven't seen it with any other. The only difference with the latest ADM is that the whole file no longer saves, the file only save to the first messed up frames.
Title: Re: Can I save this edited file some way in v2.7.5?
Post by: fish on January 13, 2020, 10:24:51 AM
Just to add some observations about editing DVB-T HD files,  I am 99% sure the problem isn't caused by ADM. For what I have experienced it is caused by wrongly ordered frames in DVB-T HD files. My heavy weight editor of choice is Avid Media Composer and when the files are linked into MC this shows up the wrongly ordered frames. When the frames are played  forward frame by frame, every 1,2,3 or 4 seconds the playback jumps 2 frames forward, one frame back and then two frames forward, equivalent to a frame order of 1,2,3,4,6,5,7,8,9. This repeats seemingly randomly every few seconds.
These wrongly ordered frames don't show up in any media player I have tried, nor does it show up in playback in ADM. When editing in ADM in copy mode, if your chosen edit points randomly happen to be on these wrongly ordered frames, the saved file have artifacts and/or a frozen section of video around those edit points. If you are lucky and your edit points are not on any wrongly ordered frames the saved file will play as though everything is fine. The wrongly ordered files will still be there but most playback methods will not show them as wrongly ordered.
Another thing I have noticed is, if you are unlucky with your edit points and get frozen video  for  maybe 5 seconds, if the play head is moved back to just after the edit, the video section that freezes will play normally. It is only when the play head passes over the edit point, the video freezes. Another oddity is that video converters re order the frames to the correct positions.
Title: Re: Can I save this edited file some way in v2.7.5?
Post by: eumagga0x2a on January 13, 2020, 11:28:54 AM
Quote from: fish on January 12, 2020, 01:13:58 AM
The problem does not arise if you simply take a whole DVB-T  HD file and save it in a mp4 container, the problem arises around edits. Try taking 10 mins of the video and remove 5 - 10, 5 second sections.

Still can't reproduce. Of course, depending on cut positions I get a warning about picture order count (POC) going back, which should be taken seriously for compatibility with ffmpeg-based video players, but apart from that: WFM.

QuoteThis has been a long running problem caused by this type of file, see the post 'Broken frames due to H.264 Open-GOP (DVB MPEG-TS)' which describes it very well.

This is all solved for H.264 streams more than half a year ago.

Quote from: fish on January 13, 2020, 10:24:51 AM
Just to add some observations about editing DVB-T HD files,  I am 99% sure the problem isn't caused by ADM. For what I have experienced it is caused by wrongly ordered frames in DVB-T HD files. My heavy weight editor of choice is Avid Media Composer and when the files are linked into MC this shows up the wrongly ordered frames. When the frames are played  forward frame by frame, every 1,2,3 or 4 seconds the playback jumps 2 frames forward, one frame back and then two frames forward, equivalent to a frame order of 1,2,3,4,6,5,7,8,9. This repeats seemingly randomly every few seconds.

Please provide a sample captured from DVB-T and the said sample edited by Avid Media Composer which allows to reproduce the issue, thanks. I don't encounter the problem you describe when using my equipment, both software- and hardware-wise.
Title: Re: Can I save this edited file some way in v2.7.5?
Post by: fish on January 15, 2020, 12:15:46 PM
Here is an example of what I am seeing and have been seeing for as long as I can remember but simply work round it. There is a difference I have noticed between v2.7.3 and v2.7.5 and that difference can be seen in the size and length of the saved file, one from v2.7.3 and the other from v2.7.5. As you can see, both files freeze at around 10s - 15s but only if the play head passes over the edit point around 10s, the video will play normally if the play head is moved to eg. 11s, this is normal.
What has changed can be seen from the edit around 39s. v2.7.3 saves the whole file before and after the 39s edit but v2.7.5 saved the video only up to that edit the rest is not saved.
I have uploaded a test file along with edit points and the file resulting from both versions. The edit points are sequential, so the position of edit 2 depends on edit 1 having been made and edit 3 on 1 and 2. Here is the link to the files.

https://file.io/kZWXfk
Title: Re: Can I save this edited file some way in v2.7.5?
Post by: fish on January 15, 2020, 12:18:42 PM
I should have also added, that a file edited in Media Composer would be of no help, as all such editors either need to convert files before editing or after editing or both, unless the codec type is native to that editor (they generally use non inter frame codecs). The resultant re encoded files would preserve the original frame order but the file would be re encoded and so defeats the purpose of using ADM. DVB-T HD files also need to be re wrapped from .ts to .mp4 before MC can use them.
Title: Re: Can I save this edited file some way in v2.7.5?
Post by: eumagga0x2a on January 15, 2020, 12:32:09 PM
Quotefile.io

404
Page not found

In doubt, please use WeTransfer, Mega, Dropbox or Google Drive – services which are proven working well.
Title: Re: Can I save this edited file some way in v2.7.5?
Post by: fish on January 15, 2020, 12:41:37 PM
OK, it worked a few minutes ago but obviously isn't now I will try another, that was 'File IO' by the way.
Title: Re: Can I save this edited file some way in v2.7.5?
Post by: fish on January 15, 2020, 12:51:26 PM
Try this.

https://file.io/rOiRUZ
Title: Re: Can I save this edited file some way in v2.7.5?
Post by: eumagga0x2a on January 15, 2020, 12:54:18 PM
Same.
Title: Re: Can I save this edited file some way in v2.7.5?
Post by: fish on January 15, 2020, 12:58:54 PM
That also failed, I'll try again with a different one.
Title: Re: Can I save this edited file some way in v2.7.5?
Post by: fish on January 15, 2020, 01:06:06 PM
This one should be ok.

https://drive.google.com/open?id=1h8mWm_un3dXbh1BnsvVFaGoLmE6w1bfd
Title: Re: Can I save this edited file some way in v2.7.5?
Post by: eumagga0x2a on January 15, 2020, 01:11:08 PM
Yes, thanks, got the sample. Will look into it ASAP.
Title: Re: Can I save this edited file some way in v2.7.5?
Post by: eumagga0x2a on January 15, 2020, 01:56:04 PM
I can reproduce the issue with the cut no. 4 from frame 1247 to 1462 (counting from zero) or 00:00:50.713 to 00:00:59.313 without previous cuts, we get identical DTS (DTS collision) out of editor. The video is a mix of fields and frames, this is indeed a blind spot in all the improvements Avidemux received last summer (all samples I can produce myself contain exclusively frames).

Will look into it, thank you for the sample.
Title: Re: Can I save this edited file some way in v2.7.5?
Post by: eumagga0x2a on January 15, 2020, 02:05:48 PM
I can also reproduce that we miss for some reason one occurence of POC going back (the problem only FFmpeg and FFmpeg-based players experience due to a questionable strategy to deal with such irregularities) after a cut and don't issue a warning.
Title: Re: Can I save this edited file some way in v2.7.5?
Post by: fish on January 15, 2020, 10:47:28 PM
The strange thing is the edit you describe,  '00:00:50.713 to 00:00:59.313 without previous cuts', completes without a problem for me, although I can't tell whether you are referring to a saved file or something in the internals of ADM.
I tried playing the sample file in Media Composer and it crashes at around 28-29 seconds. So the file is far from perfect, even though it is a typical DVB-T HD file and the problems don't show up in media players or when playing in ADM. Media Composer's playback engine is often criticised but it does seem discriminating enough to know a good video file from a bad one even though it's not really designed for inter frame codec editing)
I'm pretty sure I have seen this incomplete save with progressive DVB-T HD files as well as the MBAFF type, (movies are all broadcast in progressive), so I will try to get a suitable example.
Title: Re: Can I save this edited file some way in v2.7.5?
Post by: eumagga0x2a on January 17, 2020, 12:10:00 AM
Quote from: fish on January 15, 2020, 10:47:28 PM
The strange thing is the edit you describe,  '00:00:50.713 to 00:00:59.313 without previous cuts', completes without a problem for me

Same here, I just pointed out the source video frame numbers and timestamps of the last cut.

QuoteI tried playing the sample file in Media Composer and it crashes at around 28-29 seconds. So the file is far from perfect

The frame 737, marked as an IDR in the stream, contains an invalid bit sequence where idr_pic_id value should be encoded. This might be the issue which the decoder in Media Composer fails to handle.
Title: Re: Can I save this edited file some way in v2.7.5?
Post by: eumagga0x2a on January 21, 2020, 09:15:37 PM
Having spent quite some time on this topic, I would recommend to bite the bullet and to always re-encode such open GOP streams when deleting a middle portion.

There are fundamental problems which make it impossible to provide clean results in copy mode. The main issue is that even if we adjust segment layout to account for the presence of early B-frames (we don't do it yet, the failure to save your sample in copy mode after performing the specified cuts is the result), these frames may be (in your sample, they simply are) not cleanly decodable because they really reference frames from the previous GOP, now gone. At the same time they can't be dropped because other frames often use them as reference. So we end up with artifacts in some pictures, stutter etc. This is the best possible outcome. Is it worth the hassle?

A smaller but still complicated enough issue is that segment layout with segment boundaries at keyframe timestamps is strictly speaking invalid when we do stream copy and early B-frames are present.

Segment start must be set to the earliest non-droppable B-frame of the GOP which of course precedes the keyframe in display order but follows it in the stream order. At the same time the previous segment must be shortened to the last frame displayed before the segment switch (which is done as soon as PTS or DTS of a frame is equal or greater than the segment duration) + time increment.

These steps can be performed manually (an arduous procedure) while ignoring the warnings Avidemux will throw at the user. Doing so avoids timestamp collision and the failure to save video. We still have to put up with damaged pictures after most of the cuts.

Last but not least, the only real IDR (instant decoder refresh) frame no. 738 in the stream is advertized by the preceding SEI message wrongly as a recovery, non-IDR frame, so we distrust its IDR nature and warn about end point of the cut being set to this frame while it is the only safe place to do such a cut!

Your sample with the provided STR (steps to reproduce) helped to fix three major bugs in Avidemux and helped me to understand many aspects of the problem better, thank you very much again.