Cutting DVB ts questions

[jasum]

Hi,

can you please help me understanding Avidemux? I'd like to cut HD(mpeg4 h264) and SD(mpg) ts files (DVB recordings) with Avidemux 2.7.6 without changing the codec, with minimal re-compression and audio 100% in sync. Is Avidemux well suited for that? There are some things that are unclear to me:

Multiple chapters in the wiki are marked "Out of dates / Do not use" - what does that mean exactly, for example:

• Cutting MPEG files: "The muxing will introduce a 40(nbsp)ms shift by default." - has this been fixed? If not, does this apply to mpeg4 as well?
  
• Fixing MPEG files with ProjectX: does that mean that Avidemux can fix mpegs on its own now/has ProjectX become superfluous?
  
•     I've heard that some ts files contain padding and that ProjectX can remove it. Can Avidemux do that, too?
  
•     The Editing MPEG capture (DVB or IVTV) chapter strongly suggests using ProjectX, so it is still needed? So far I only used it for SD - does it work for HD(mpeg4 h264), too?

• In Cutting: there is this sentence: " There are two ways of dealing with that: " but then only one way is described. The missing second way was "You use the smart copy mode. Only available for Divx3 and mpeg4" (still found here http://avidemux.sourceforge.net/doc/en/cutting.xml.html) - has SmartCopy been removed or has the limitation to divx3/mpeg4 been removed or was omitting SmartCopy just an oversight?
  
• What about the mp4v2 "Optimize for Streaming" option? There seems to be no documentation for it - should I use it if I plan to watch the cut videos over LAN or is this only intended for internet video streaming plattforms?

[eumagga0x2a]

I'd like to cut HD(mpeg4 h264) and SD(mpg) ts files (DVB recordings) with Avidemux 2.7.6 without changing the codec, with minimal re-compression and audio 100% in sync.

Is "minimal re-compression" really what you want? Any re-encoding with a lossy codec takes time and impacts quality. The main purpose of Avidemux is cutting in copy mode, avoiding re-encoding. Especially for copy mode, but also for a lot of other reasons I would recommend to use a more recent build as 2.7.6 suffered from some substantial mistakes in code which checks cut points in H.264 and HEVC streams for viability. These bugs were fixed shortly after the release.

Multiple chapters in the wiki are marked "Out of dates / Do not use"

Almost the entire wiki is hopelessly out of date, stuck in 2.5.x or early 2.6.x era. It is valuable as a document of history though.

A/V sync is usually not an issue with DVB-T/C/S MPEG-TS files without defects nowadays. Avidemux won't fix any defects (it may skip a brief stretch of garbage data depending on particular situation). I personally didn't have a case where I would need ProjectX during the last decade or even longer. Additionally, ProjectX operates on MPEG-2 video only – most worthwhile broadcasts use H.264 or even HEVC.

I've heard that some ts files contain padding and that ProjectX can remove it. Can Avidemux do that, too?

The MPEG-TS demuxer in Avidemux picks only packets belonging to PES (packetized elementary streams) it cares about. Everything else is discarded. Within elementary streams, in H.264 and HEVC access units, filler NAL units, used to keep bitrate high enough even when the picture slice in the access unit is very small (e.g. a B-frame), are discarded when saving the stream in copy mode.

"Smart copy" was removed a very long time ago. It is not viable with modern codecs and open-GOP type of H.264 and HEVC streams used in broadcasts.

What about the mp4v2 "Optimize for Streaming" option?

Don't use MP4v2, it is a legacy deprecated muxer. If you want to mux into mp4, please use the current, maintained, (almost) up-to-date libavformat-based MP4 muxer.

When creating a MP4 file, the index (the structure containing the information about offsets of individual chunks in the file and their timing) can be created only after all the data has been written. This creates a problem that in order to be able to know what is in the file, an application needs to read data from the end of it. When playing such a file over network, the application needs to be able to seek to the end before playback can be started. This is not always possible, many streaming devices will send data only sequentially. To help with that, the file can be completely rewritten in a separate step after the actual muxing has been finished, moving the index to the beginning of the file.

The MP4 muxer offers alternatively a second, different strategy – fragmentation. This puts each GOP into a sort of a small MP4 file within a single container, avoiding the need for time consuming and drive churning rewrite of the entire file. The problem with that is spotty support for this type of input among video players.

Whatever strategy you choose – non-optimized files won't play over network without seek capability before the entire file has been transmitted, which defies the purpose of streaming.

[jasum]

Hi eumagga0x2a,

Is "minimal re-compression" really what you want?

by "minimal re-compression" I meant something like SmartRendering - only re-encoding selectively if there isn't an Iframe at the position where I need the cuts to be for a seamless result.
So can't I achieve a seamless cut in Avidemux without re-encoding the whole movie or is the requirement for cutting on Iframes also outdated? Why wasn't SmartRendering kept for cutting mpeg2?

Additionally, ProjectX operates on MPEG-2 video only - most worthwhile broadcasts use H.264 or even HEVC.

Over here, most of free-tv is still sending in SD, so I have to handle that stuff somehow. So far I was using ProjectX+Cuttermaran/Imago - it worked really well but was also a tedious, time-consuming process and Cuttermaran is windows only, that is no longer an option.
Would be great if Avidemux could replace that pipeline completely.

Version 2.7.6 is the latest on my Linux, thanks for the heads-up! So I will use git or an appimage - what is the difference between appImage, appImage4, appImage4Buster and appImage_support in https://avidemux.org/nightly/?

The MPEG-TS demuxer in Avidemux picks only packets belonging to PES (packetized elementary streams) it cares about. Everything else is discarded. Within elementary streams, in H.264 and HEVC access units, filler NAL units, used to keep bitrate high enough even when the picture slice in the access unit is very small (e.g. a B-frame), are discarded when saving the stream in copy mode.

That's the first solid info I got on this topic in years, thank you so much!

[eumagga0x2a]

So can't I achieve a seamless cut in Avidemux without re-encoding the whole movie or is the requirement for cutting on Iframes also outdated?

The requirement to start a segment at a keyframe is based on maths, not specific to any particular application.

In reality, the situation is more complicated, at least for H.264 and HEVC streams, because we have to deal with open-GOP type of streams and keyframes in such streams are not IDR (instantaneous decoder refresh) but only so-called recovery frames (a frame, where a decoder can begin decoding the stream and output a full picture a number of frames later). Internal counters in slice headers are not reset at such keyframes and cutting away an inner portion of video may result in incompatible counter values meeting each other (Avidemux is now able to warn about such conflicts, the detection was partially broken in 2.7.6 but fixed shortly thereafter).

Why wasn't SmartRendering kept for cutting mpeg2?

I don't know, this all happend long before I started to contribute to the project. Probably too complicated with little benefit as GOP size in MPEG-2 is rather short.

So far I was using ProjectX+Cuttermaran/Imago - it worked really well but was also a tedious, time-consuming process and Cuttermaran is windows only, that is no longer an option.
Would be great if Avidemux could replace that pipeline completely.

I was using ProjectX + Cuttermaran back then too, switched to Avidemux alone a decade or so ago due to Windows becoming irrelevant for me. There were some improvements regarding handling of interlaced video recently, but I don't plan to work on any MPEG-2 related features.

So I will use git or an appimage - what is the difference between appImage, appImage4, appImage4Buster and appImage_support in https://avidemux.org/nightly/?

If your Linux distribution has glibc in version 2.28 or later, please use appImages (preferably the most recent one) which mention GLIBC in their name from the "appImage4Buster" directory – these are new-style appImages with better hardware acceleration and codec support for more up-to-date Linux distributions, which also allow to launch Jobs GUI and Avidemux CLI application from the appImage. Else you are restricted to legacy appImages from the "appImage4" directory. Ignore all others.

If you plan to follow closely Avidemux development, you might want to build Avidemux yourself from git master and run it without installation from the build directory (as I do).

[jasum]

So I'm using the latest appimage now, it worked well for the first movie, but the second one is giving me trouble:

Cut a commercial that luckily had i-frames at start and end, then tried cutting a second one also from i-frame to i-frame but got this message: "This video uses non-IDR recovery points instead of IDR as keyframes. Picture reordering information in the video stream is not reset at non-IDR frames. The chosen start an end points of the deletion may result in playback interruption due to reversed display order of frames if saved in copy mode. Proceed anyway?"

I thought from I-frame to I-frame always works?

[eumagga0x2a]

I thought from I-frame to I-frame always works?

No, the message tries to condence a lengthy explanation into few words. IDR to IDR always works. Keyframes which Avidemux calls "I-frames" are not necessarily IDR (instantaneous decoder refresh) frames, and there are hardly any IDR frames in H.264 and HEVC streams used by broadcasters. Strictly speaking, any deletions in the middle of such streams creates invalid output, but most players exhibit little to no problems handling them.

There is one notable exception: all FFmpeg-based players and editors don't handle well the case described in the warning. This problem is specific to FFmpeg, but because of importance and popularity of applications like mpv and VLC, recent versions of Avidemux implement a check for POC going back at cut points.

If you don't need to play the saved video with VLC & Co, you can almost safely ignore the warning, else you need to try to find another combination of start and end point for deletion.

Avidemux works around the FFmpeg problem by jumping to the next keyframe when it gets stuck at the cut point.

[jasum]

Well, I'm mostly using VLC and mpv, so I guess I'm out of luck again.

But I can always cut away stuff at the beginning and the end without any problems?
Can't Avidemux show if a keyframe is IDR, that way I don't have to start encoding to know if it works.

[eumagga0x2a]

But I can always cut away stuff at the beginning and the end without any problems?

Yes, but instead of thinking "cut away" think and do "save from A to B".

Can't Avidemux show if a keyframe is IDR, that way I don't have to start encoding to know if it works.

When copy mode is chosen, Avidemux checks cut points during editing and displays a warning, exactly in order to catch such issues before the user starts saving video (if a user chooses to discard such warning, no more warnings of the type are shown for this video during editing).

Avidemux prints results of cut checks incl. IDR / non-IDR to its log file or to stdout on Linux. Regading DVB streams, you can safely assume that IDR are inserted only when there is no other way, e.g. on format changes (which we absolutely don't support) or when encoding mode changes. I don't think it makes sense to bother user with these aspects beyond the warning you cited.

[jasum]

Regading DVB streams, you can safely assume that IDR are inserted only when there is no other way, e.g. on format changes (which we absolutely don't support) or when encoding mode changes.

In the case of an h264 ts file with 3 commercial breaks, I could delete the first and the last one without a warning. Only deleting the commercial break in the middle resulted in the IDR message (I double checked).
It was still the case after cutting away stuff at start and end of the video, saving as mkv and loading it back into avidemux. There was no mention of IDR in the terminal output when deleting the first and last commercial break.

[eumagga0x2a]

The warning is shown only if a really big trouble is detected. If there was no mention of IDR in the terminal output, the keyframes were merely recovery (i.e. direct access) frames.

[jasum]

Thank you!