Trimming/Cutting bug (corrupted frames are added before the first I-frame)

[Arvy]

I'm trying to cut some HDTV files. They are interlaced h264+AAC in .trp container (I've tried remuxing to .ts or .mkv first with the same result). I don't want it to be reencoded, so I choose copy for both video and audio and save it to .ts container (tried .mkv as well with the same result).
I'm aware of the I-frame cutting concept, so I'm putting A mark at I-frames only, but the actual cutting seems to happen several frames before the specified I-frame, resulting in several corrupted frames and unneeded audio in the beginning of the new file. The end of the file is cut perfectly.
No matter where exactly I put the A mark for cutting, even if I put it at B or P frames, it cuts at the closest I-frame minus several frames, which results in corrupted beginning.
Funny thing is, if I open the resulting file (with corrupted frames at the beginning) and save it without doing anything else (or choose the first I-frame as a beginning point), the corrupted frames are gone and I get the result I wanted from the beginning (well, almost).

Examples at google drive https://drive.google.com/open?id=1wVRyhDzVPp9hCSE26ufYhfBPC0LbWxVl
test_uncut.ts = the sample file cut with AviDemux from the bigger .trp file for sampling purpose.
test_cut1.ts = cut with AviDemux from 00:00:07.907 to 00:00:19.252 of the sample file. Extra unwanted frames and audio are added to the beginning (before the specified I-frame).
test_cut2.ts = test_cut1.ts opened in AviDemux and saved without any changes. Extra frames from test_cut1.ts are gone!

I'm running Win7x64 and I've tried AviDemux 2.6.12, 2.6.21, 2.7.0 r170429 and 2.7.0 r171206 with the same result.

I'm not sure what exactly causes the problem and where to look for the solution, but it happens with all files produced by my HDTV stb.

Any ideas?

[Jan Gruuthuse]

- mpeg-ts .trp format to .ts did you do this in avidemux? If not can you provide  an original sample recording 1:00 minute duration?
- HD stb? What is the origin of the recording (DVB-T/T2 or C/C2 or S/S2)?
- While switching between avidemux versions, you did delete the .idx2 files related to these recordings?

[Jan Gruuthuse]

There seems to be an issue:
I can set a marker  [A ] @ 00:00:07.907 I-TFF (02)
I can't select marker [ B] @00:00:19.252
available is only 00:00:19.386  I-TFF (02) or 00:00:19.919 I-TFF (02)

While only using copy mode, you would need to use only editing points select with up/down keyboard arrow

Tested with avidemux (commit d2fe759) 2.7.0 171206-64bit Ubuntu HWE Xenial with x265 version 1.9

[Arvy]

- mpeg-ts .trp format to .ts did you do this in avidemux? If not can you provide  an original sample recording 1:00 minute duration?

Yes. I did it in AviDemux. What do you want me to use for cutting? I think I don't have any software that can cut and save files as .trp instead of muxing it as .ts. I have a different file directly from stb which is 628mb (totally untouched), would that be too much? It's uploaded here: https://drive.google.com/open?id=1mvRqEM5SxyotCasZZ6kcePFHxC8qTLiC

- HD stb? What is the origin of the recording (DVB-T/T2 or C/C2 or S/S2)?

DVB-S2

- While switching between avidemux versions, you did delete the .idx2 files related to these recordings?

Yes, I deleted it every time.

There seems to be an issue:
I can set a marker  [A ] @ 00:00:07.907 I-TFF (02)
I can't select marker [ B] @00:00:19.252
available is only 00:00:19.386  I-TFF (02) or 00:00:19.919 I-TFF (02)

While only using copy mode, you would need to use only editing points select with up/down keyboard arrow

Tested with avidemux (commit d2fe759) 2.7.0 171206-64bit Ubuntu HWE Xenial with x265 version 1.9

00:00:19.252 is a P-frame, I know, but it's the end, not the beginning, so it should be OK to use. I can select it no problem. In my case the end is cut perfectly with frame precision no matter where you put the marker, be that I-frame or not. The beginning gets corrupted no matter what though, even when I close the selection with an I-frame.

Do you get no corrupted frames at the beginning with your setup?

PS. Thanks for looking into it.

[Jan Gruuthuse]

downloaded, took some minutes.

[Jan Gruuthuse]

sample cut
Mark [A ] @ 00:00:49.282
Mark [ B] @ 00:00:59.025

Space_Shower_HD-cut.ts
Download link  https://we.tl/LJGYSar801
1 file, 7.32 MB in total ・ Will be deleted on 14 December, 2017
------------
that seemed to go well, played back with vlc and avidemux (keep in mind avidemux is not a media player).
------------
Developers please have a look at this, when loaded into avidemux

The time index shows 1st frame as 00:00:00.166 Space_Shower_HD-cut.ts

[Arvy]

Now that you mention VLC... It plays .ts files totally different for me (I have a very old version though, 2.1.5), always starting with grey artifacts video (no matter what .ts file I drop to it), but I don't hear the extra sound. AviDemux doesn't show extra frames and audio as well (and therefore cuts it perfectly on the second try?).
I use MPC (With LAV filters) and it shows several broken frames with previous GOP audio for your cut as well. PotPlayer and Windows Media Player do the same as MPC. Opening it with any source filter in avisynth show extra frames and sound as well.
But, if I load your cut file (7.32MB) to AviDemux and just save it, it becomes 7.14MB and starts to work perfectly with any player. Magic.

[Jan Gruuthuse]

VLC 2.2.2 Weatherwax was used here.
See if I can do some more testing here later today.
Forgot to mention this was .trp edited, save marked positions.
Could be to do with the source video has a
Frame rate                               : 29.970 (30000/1001) fps (NTSC/60Hz)

Code Select Expand

General
Complete name                            : /home/jan/Downloads/Space_Shower_HD.trp
Format                                   : MPEG-TS
Format profile                           : No PAT/PMT
File size                                : 628 MiB
Duration                                 : 17mn 4s
Overall bit rate mode                    : Variable
Overall bit rate                         : 5 140 Kbps

Video
ID                                       : 4098 (0x1002)
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : Main@L4
Format settings, CABAC                   : Yes
Format settings, ReFrames                : 4 frames
Duration                                 : 17mn 4s
Bit rate mode                            : Variable
Maximum bit rate                         : 20.0 Mbps
Width                                    : 1 440 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate                               : 29.970 (30000/1001) fps
Standard                                 : Component
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Interlaced
Scan type, store method                  : Separated fields
Scan order                               : Top Field First
Color range                              : Limited
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709

Audio
ID                                       : 4162 (0x1042)
Format                                   : AAC
Format/Info                              : Advanced Audio Codec
Format version                           : Version 2
Format profile                           : LC
Muxing mode                              : ADTS
Duration                                 : 17mn 4s
Bit rate mode                            : Variable
Channel(s)                               : 2 channels
Channel positions                        : Front: L R
Sampling rate                            : 48.0 KHz
Frame rate                               : 46.875 fps (1024 spf)
Compression mode                         : Lossy
Delay relative to video                  : -616ms

Hope developer(s) pick up on this.

[Jan Gruuthuse]

load Space_Shower_HD-cut.ts (1st edit) 7.7MB, do nothing else but:
save straight away as Space_Shower_HD-cut_2ndsave.ts
7.5MB drops 67ms (9s 743ms - 9s 676ms)

Code Select Expand

General
ID                                       : 1 (0x1)
Complete name                            : /home/jan/Videos/Space_Shower_HD-cut.ts
Format                                   : MPEG-TS
File size                                : 7.32 MiB
Duration                                 : 9s 743ms
Overall bit rate mode                    : Variable
Overall bit rate                         : 6 186 Kbps

Video
ID                                       : 256 (0x100)
Menu ID                                  : 1 (0x1)
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : Main@L4
Format settings, CABAC                   : Yes
Format settings, ReFrames                : 4 frames
Codec ID                                 : 27
Duration                                 : 9s 760ms
Bit rate mode                            : Variable
Maximum bit rate                         : 20.0 Mbps
Width                                    : 1 440 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate                               : 29.970 (30000/1001) fps
Standard                                 : Component
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Interlaced
Scan type, store method                  : Separated fields
Scan order                               : Top Field First
Color range                              : Limited
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709

Audio
ID                                       : 257 (0x101)
Menu ID                                  : 1 (0x1)
Format                                   : AAC
Format/Info                              : Advanced Audio Codec
Format version                           : Version 4
Format profile                           : LC
Muxing mode                              : ADTS
Codec ID                                 : 15
Duration                                 : 9s 831ms
Bit rate mode                            : Variable
Channel(s)                               : 2 channels
Channel positions                        : Front: L R
Sampling rate                            : 48.0 KHz
Frame rate                               : 46.875 fps (1024 spf)
Compression mode                         : Lossy
Delay relative to video                  : -167ms

Menu
ID                                       : 4096 (0x1000)
Menu ID                                  : 1 (0x1)
Duration                                 : 9s 743ms
List                                     : 256 (0x100) (AVC) / 257 (0x101) (AAC)
Service name                             : Service01
Service provider                         : FFmpeg
Service type                             : digital television

Code Select Expand

General
ID                                       : 1 (0x1)
Complete name                            : /home/jan/Videos/Space_Shower_HD-cut_2ndsave.ts
Format                                   : MPEG-TS
File size                                : 7.14 MiB
Duration                                 : 9s 676ms
Overall bit rate mode                    : Variable
Overall bit rate                         : 6 152 Kbps

Video
ID                                       : 256 (0x100)
Menu ID                                  : 1 (0x1)
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : Main@L4
Format settings, CABAC                   : Yes
Format settings, ReFrames                : 4 frames
Codec ID                                 : 27
Duration                                 : 9s 643ms
Bit rate mode                            : Variable
Maximum bit rate                         : 20.0 Mbps
Width                                    : 1 440 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate                               : 29.970 (30000/1001) fps
Standard                                 : Component
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Interlaced
Scan type, store method                  : Separated fields
Scan order                               : Top Field First
Color range                              : Limited
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709

Audio
ID                                       : 257 (0x101)
Menu ID                                  : 1 (0x1)
Format                                   : AAC
Format/Info                              : Advanced Audio Codec
Format version                           : Version 4
Format profile                           : LC
Muxing mode                              : ADTS
Codec ID                                 : 15
Duration                                 : 8s 746ms
Bit rate mode                            : Variable
Channel(s)                               : 2 channels
Channel positions                        : Front: L R
Sampling rate                            : 48.0 KHz
Frame rate                               : 46.875 fps (1024 spf)
Compression mode                         : Lossy
Delay relative to video                  : 811ms

Menu
ID                                       : 4096 (0x1000)
Menu ID                                  : 1 (0x1)
Duration                                 : 9s 676ms
List                                     : 256 (0x100) (AVC) / 257 (0x101) (AAC)
Service name                             : Service01
Service provider                         : FFmpeg
Service type                             : digital television

[AQUAR]

Just a loose comment:
Since this is interlaced video and IIRC, mean has mentioned that one field might be I and the other field might not be I.
So cutting on an a frame labelled "I" by ADM may in fact contain a predictive component.
I don't think ADM knows how to deal with interlaced AVC very well, and maybe behind why you get some corruption.

[fish]

I have posted on this issue before, it happens with DVB-T HD files. When using the up/down or gui icons to move to I frames, the cut is always 2 frames out. Let's say we have frames 1-2-3-4-5-6-7-8-9 chosen in the middle of a longer file and the I frame is 4. When the in point of a remove edit is set to 4, frames are removed from frame 2 until the out point. Sometimes there is corruption from the out point on, sometimes there is also a frozen GOP following the edit and sometimes not. All you can do is move to the next I frame and begin the cut from there. The strange thing is that from trying different editing points, it seems to me that the problem appears to be caused not by the I frame just after the remove edit but the frames in front of the 'in'edit point. I have dismissed the random freezing and corruption as just a symptom of broadcast glitches but the 2 frame inaccuracy of the in point is real.
One more thing I have noticed, again with DVB-T HD files, I have a few files from 2012 that I compared with recent broadcast files. When I move to a scene change in the 2012 files, they are exactly on an I frame but with recent files the I frame is exactly 2 frames after a scene change. So something appears to have changed in the later broadcast files to cause the issue. If the I frames are no longer exactly on a scene change it makes the editing a bit more clumsy, as we must either lose 2 frames or include an extra group GOP after the scene change we wish to cut from.
What is counter intuitive is, if the I frame is two frames after the scene change and we move to that I frame as the start of a remove edit, we should expect to see 2 frame after a scene change, remaining in the edited clip. Instead there is 2 frames missing before the scene change. I am using scene change points as an example because it is easier to spot, but 2 extra frames are removed whether the edit is at a scene change or not.

[AQUAR]

Some of these DVB_T broadcasts would be encoded using MBAFF where macroblocks in a frame/field can be either progressive or interlaced.
It all just means that AVC is a difficult codec when it comes to "copy edits".

One day maybe the Authors will re-introduce "smart copy" code that will get rid of such issues.

[eumagga0x2a]

Is Space_Shower_HD.trp the file supposed to cause issues when saved in copy mode with cut points on I-frames? Works perfectly fine for me on a quick test, no corruption (missing ref frames) in Avidemux, ffplay, mpv and QuickTime.

[Jan Gruuthuse]

source file Space_Shower_HD.trp possible cause: yes (interleaved frame order perhaps? <- not certain about this, wild guess)
1st edit:
- Mark [A ] @ 00:00:49.282
- Mark [ B] @ 00:00:59.025
save as new video 1 mpeg-ts (copy mode)

load video 1

save as new video 2 mpeg-ts (copy mode)

Notice the file size difference. between video 1 and video 2.
The video 2 plays without errors on older VLC 2.1.5.
Newer VLC 2.2.2 Weatherwax seems to cope with video 1 garbage in front?

[eumagga0x2a]

I used the MP4 muxer during my testing in order to be able to play the result in QuickTime. Will try to reproduce the issue with the MpegTS muxer later.