Failure saving damaged MPEG-TS file in copy mode

[AlexBelz35]

Hello @eumagga0x2a

I'm a professionnal of live Video Broadcasting (TS).

I'm also a periodic user simple video editing tools, and here I wanted to try (again) AviDemux.
My goal is to edit Terrestrial DVB-T TS files (single TV program inside, with several audio tracks) recorded by a PVR (Etimo 2).
Well known issue is that the TS stream sometimes features loss of packets (due to "normal" RF perturbance).
Many TS players are able to cope with such mising packets on a TS file, they will just try their best, often showing picture artifacts during a GOP (typ. 1 second). This is considered acceptable, as long as the video keeps playing normally from the next I picture.

Here nevertheless, when I trimm & export the TS sample file below, I get AviDemux export crashing at 38% due to time discontinuity (indeed, there may be a loss of packet leading to some TS Countinuity Counters issues and as a consequence, PTS time jumps).
I tried with the official 2.8.1 and the nihtly  2.8.2 ; same issue.

Please could you identify the issue and imagine a way to go over this during the export processing ?
I don't expect AviDemux to correct anything, just to continue the cut & export process. Up to the TS player then to manage this (normal) TS packet disruption in the file.

Please find below some files to help reproducing & pinpoiting the issue :
  https://1drv.ms/f/s!Ahp2yXCYf8WmrxOHnstsOfUoIffI?e=rTuEcX    (shared onedrive folder, readonly)

• cmd_merged.ts : the original TS file
• avidemux_test_cmd_merged.py : the AviDemux project
• Bug38%.png : screen capture of the crashing export
• Bug38%_.png : other screen capture of the crashing export
• admlog.txt : AviDemux log file

Can this help ?
Tell me if you need something else.

Warmest regards,
Alexandre

[eumagga0x2a]

Hello,

Thank you for having tried the latest(?) nightly, this is generally the right thing to do before asking questions in the forum. I've detached your post as it doesn't contain anything related to current development.

If we look at the frames between 90000 and 90100 where the failure occurs

Code Select Expand

ed = Editor()
for frame in range(90000, 90100):
    ed.printFrameInfo(frame)
we get

Code Select Expand

Frame 90027 Flags 4000 (B/F) DTS 00:59:11,249 PTS 00:59:11,289 / 00:59:10,440 Size: 9025
Frame 90028 Flags 4000 (B/F) DTS 00:59:11,289 PTS 00:59:11,409 / 00:59:10,560 Size: 18394
Frame 90029 Flags 4000 (B/F) DTS 00:59:11,329 PTS 00:59:11,369 / 00:59:10,520 Size: 9160
Frame 90030 Flags 4000 (B/F) DTS 00:59:11,369 PTS 00:59:11,449 / 00:59:10,600 Size: 5726
Frame 90031 Flags 4000 (B/F) DTS 00:59:11,249 PTS 00:59:11,289 / 00:59:10,440 Size: 9025
Frame 90032 Flags 4000 (B/F) DTS 00:59:11,289 PTS 00:59:11,409 / 00:59:10,560 Size: 18394
Frame 90033 Flags 4000 (B/F) DTS 00:59:11,329 PTS 00:59:11,369 / 00:59:10,520 Size: 9160
Frame 90034 Flags 4000 (B/F) DTS 00:59:11,369 PTS 00:59:11,449 / 00:59:10,600 Size: 9818
This means that frames 90031 - 90034 are obviously a duplicate of the stretch of the file with frame 90027 - 90030 (either the initial frame 90030 was truncated or the second one contains some garbage at the end, didn't try to figure this out so far). How should an application know how to handle this brokenness?

To be precise, export doesn't crash. It stops once libavformat rejects the next frame Avidemux feeds to it.

[alexstorm]

I wonder if the original merged .ts file has issues.  I notice that when .ts files are streamed, often a single file is skipped or damaged due to online traffic or server issues.  Many 3rd party video players just skip over the missing data and the user experiences a 7 second jump, which almost everyone would not notice.  However, it the merged ts file has this corrupt data in it and you are trying to edit, you obviously can run into edit errors.

Whenever I deal with .ts video, I make sure that all individual ts files are accurate and none are missing.  Then I assemble the merged file myself from these quality tested individual files.  I make immediately a .mp4 merged file.  Editing any of these have never caused an issue with AVIDemux.

When you test an issue with the software, use a proven demo file that has been professionally created to a size and codec and already valid as a test video.  Otherwise you are using media, that may have issues.  Playback in VLC does not prove that the file is valid.  You can run a test with ffmpeg.  If you test with random media, how are you suppose to pinpoint the issue that causes the problem?

[eumagga0x2a]

I wonder if the original merged .ts file has issues.

Yes, it has. Reading the topic prior to replying is very much appreciated, thank you.

[AlexBelz35]

Hi

Thank you very much guys for your feedbacks.

Strange that TS paquets are duplicated ; I would think i would get only losses of packets with terrestrial RF input.
Nevertheless as the source TS file was merged from multiple Etimo TS files (many PVRs do this to cope with the legacy FAT size limitations), the PVR may leave some duplicate TS packets at the boundaries of the individual files. That is probable.

As my merge workflow is a basic xcopy, there is no coherency analysis nor duplicate removal.

Do you know a (free) tool that would you recommend that would merge several TS files, removing duplicate packets ?

Note that I agree VLC does not give a proof of compliance of my merged TS file. I just meant that it succeeds to skip over the error. Brillant.

Best regards,
Alexandre

[alexstorm]

Alex,
your merged .ts files on one drive has corrupted time line data.  It is not showing an end time using VLC.  There are alternate audios and several other additional files in a 7.3GB download for a 1080P playback.  Doing anything with this file in Windows starts the virus check.  I cannot even rename the file.

I think you are using a media copy that is too bloated and damaged.  This is causing your problem.  You need to grab the media in a different way.  I think trying to deal with this file is hopeless.

Related?
https://forum.videolan.org/viewtopic.php?f=2&t=55779