Notify about appended video incompatibility early on

[x2abc33d]

Hi everyone!

I've been trying to split a video into many parts, edit some of the parts with a different video editor and put all of the parts back together with Avidemux and came up with a suggestion for a UI improvement that I think might help others:

Avidemux already detects when an appended video's codec settings don't match the existing video. However, it would be nice if Avidemux would notify the user early on; right when a user tries to append a new video, not wait till the saving time, which may be much later. Even if not all settings can be validated at append time, it would still be helpful to notify the user of any problems early on. I think it's still important to check for incompatibility later on if it's possible to make some edits that would result in a broken output video.

On a related note, it would be nice if more detailed information about the incompatible settings is shown right in the UI. Currently the most general approach for a user who is not familiar with Avidemux' code is to grep the source code for "Codec or codec settings across the cut" then look for every place where ADM_EDITOR_CUT_POINT_MISMATCH is used and look for any error message prints around those lines that may be printed in the actual log file. Instead it would be hugely helpful if the GUI would simply listed the properties of the video that didn't match. eg:

Attempting to append a video with codec settings that do not match the project.
(the following could be hidden behind some checkbox in the dialog window by default)
Details:
The following codec settings detected for file <filename> do match the codec settings already present in the project:
hasPocInfo: 0 (project has: 1)
log2MaxPocLsb: 4 (project has: 8 )
refFrames: 5 (project has: 3)

As far as I can tell as of the 2.7.5 release even the log file doesn't provide this kind of information so this would be a pretty big improvement IMHO.


BTW thanks for making this very useful piece of software!

[eumagga0x2a]

However, it would be nice if Avidemux would notify the user early on; right when a user tries to append a new video, not wait till the saving time, which may be much later.

A valid suggestion if "copy" is selected. Different from other cases, this notification should be just informational as opening a file can be very time consuming, and users sometimes append first, think about the desired output mode (re-encoding vs copy) later.

On a related note, it would be nice if more detailed information about the incompatible settings is shown right in the UI. Currently the most general approach for a user who is not familiar with Avidemux' code is to grep the source code for "Codec or codec settings across the cut" then look for every place where ADM_EDITOR_CUT_POINT_MISMATCH is used and look for any error message prints around those lines that may be printed in the actual log file. Instead it would be hugely helpful if the GUI would simply listed the properties of the video that didn't match. eg:

Attempting to append a video with codec settings that do not match the project.
(the following could be hidden behind some checkbox in the dialog window by default)
Details:
The following codec settings detected for file <filename> do match the codec settings already present in the project:
hasPocInfo: 0 (project has: 1)
log2MaxPocLsb: 4 (project has: 8 )
refFrames: 5 (project has: 3)

IMVHO this information is completely unhelpful for almost all users as anyone not reading Avidemux source code would have no clue what it all means. From the ego-POV, I would have had no idea before I started to work on this feature and was forced to touch just the surface of H.264 complexity.

As far as I can tell as of the 2.7.5 release even the log file doesn't provide this kind of information

This debug message was added on July 25, 2019 by [editor/copy] Log all relevant mismatched SPS info fields for the sake of clarity, so it should be present in 2.7.5 (which is 2.7.4 with build env only fix).

[x2abc33d]

A valid suggestion if "copy" is selected. Different from other cases, this notification should be just informational as opening a file can be very time consuming, and users sometimes append first, think about the desired output mode (re-encoding vs copy) later.

I'm not familiar with the code, does append not open the file first?

As far as I can tell as of the 2.7.5 release even the log file doesn't provide this kind of information

This debug message was added on July 25, 2019 by [editor/copy] Log all relevant mismatched SPS info fields for the sake of clarity, so it should be present in 2.7.5 (which is 2.7.4 with build env only fix).

What I mean is that the log doesn't show the actual values that differ. Eg: 0 vs 1, 4 vs 8

[eumagga0x2a]

does append not open the file first?

It does, so discarding the operation because of codec unintentionally left at "copy" would be a bad idea. The problem may arise for command line usage as for compatibility reasons it should not matter whether --append follows --video-codec or vice versa.

Of course, there is also the gap of a check for audio tracks compatibility completely missing.

What I mean is that the log doesn't show the actual values that differ. Eg: 0 vs 1, 4 vs 8

The problem is how to match a set of values to a particular video so that the user understands which is which. Just printing values is easy.

[odin]

[...] The following codec settings detected for file <filename> do match the codec settings already present in the project:
hasPocInfo: 0 (project has: 1)
log2MaxPocLsb: 4 (project has: 8 )
refFrames: 5 (project has: 3)[/tt]

IMVHO this information is completely unhelpful for almost all users as anyone not reading Avidemux source code would have no clue what it all means. From the ego-POV, I would have had no idea before I started to work on this feature and was forced to touch just the surface of H.264 complexity.

Unfortunately I've to agree eumagga0x2a. And it's even much worse:
There are several ways available to analyse H.264 elementary streams (e.g. h264_parse), in order get the requested details (Sequence parameter set & Picture parameter set).
But: even if knowing these parameters, there's no mapping guide available how to set e.g. x264 options in order to achieve the desired SPS & PPS values. And from my experience that's not only a documentation issue, but it's probably also a problem of missing options and dependencies between options and also some non-deterministic behavior (depending on video content).
To make it short: I didn't managed to encode a compatible video for concatenation (here is another example).

But there's a "workaround" based on H.264 standard:
In addition to the fields SPS & PPS, there're two fields SPS-id & PPS-id (values 0...31), which are references to the SPS & PPS values. Therefore an elementary stream can contain up to 32 differently encoded videos, as long the the IDs are also different. I guess it looks like this:

• video 1 header: SPS-id #0 := <SPS details>, PPS-id #0 := <PPS details>
• video 1 data, using SPS-id #0 + PPS-id #0
• video 2 header: SPS-id #1 := <SPS details>, PPS-id #1 := <PPS details>
• video 2 data, using SPS-id #1 + PPS-id #1

Using x264 it's possible to create a H.264 file with a dedicated SPS-id & PPS-id via option "--sps-id <n>".

And using mp4box it's possible to concatenate such files (i.e. leaving the SPS-id & PPS-id unchanged):
MP4Box "part1.mp4" -force-cat -cat part2.mp4 -cat part3.mp4 -tmp . -out concat.mp4

Unfortunately avidemux seems to reset SPS-id & PPS-id to 0, i.e. avidemux cannot be used for concatenation.
Also I didn't found a way to change SPS-id & PPS-id of existing videos without re-encoding (e.g. no ffmpeg bitstream filter).

My suggestion:
Instead of requesting to output SPS + PPS details for "incompatible" videos, it would be much more beneficial if avidemux is able to simply increase SPS-id & PPS-id by 1, when it detects different SPS + PPS values in an added video (or to make it simpler: each added video always gets an incremented SPS-id & PPS-id, allowing up to 32 video-parts).

[eumagga0x2a]

All locations in Avidemux code which handle SPS and PPS (like the recently added SPS checks in MpegTS, Mp4 and Matroska demuxers as well as POC check in the editor, but also older code which creates AVCC type extradata from in-band SPS and PPS) simply assume that only one of each is present at a time and the ids are equal zero. The most recent addition – the ability to deal with in-band SPS also in AVCC type H.264 streams which normally use a single global header – exhibits the same limitation regarding SPS id. I haven't encountered a single real-life sample which uses multiple SPS yet.

Handling multiple SPS ids in all these checks would massively increase complexity, which is already unwieldy, while changing SPS or PPS id in existing stream means rewriting all slice headers.

Unless mainstream broadcasters start to use these features widely, I hope to be able to cut this corner safely.

[odin]

Just to be sure that you understood me correctly:
My primary intention is not, that avidemux is able to read and "correctly" process input-SPS-/PPS-IDs> 0.
Instead it's about the usecase from the initial post:

• The user has multiple videos, all H.264 with e.g. 720p 16x9 30fps Baseline Level X.
• The user would like to perform a lossless concatenation, i.e. add those videos to avidemux and then save in copy-mode.
• Unfortunately, some bits in SPS/PPS are different in some videos, so that result file has playback issues.
• Possible reasons for the different SPS/PPS bits: user would like to add his own intro/title image to a recording. Or user would like to concatenate "old" and "new" files, and there was some S/W update in the meantime, so that new files are a little bit different to the old ones. Or user would concatenate videos from different broadcasters/sources.

I didn't found any solution for this problem with existing software :-(
I even looked at the source code of an old ffmpeg bitstream filter for H.264, in order to change the code so that I'm able to modify the SPS-/PPS-IDs on the fly. But that seems to be far above my knowledge :-(

And the workaround I described in my prev. post (to create IDs> 0) requires re-encoding of one of the input videos with x264, i.e. that is not lossless, and therefore not a real solution.

Therefore the usecase regarding avidemux would be:

• User adds multiple videos to avidemux (all have ID=0)
• User saves them in copy-mode
• Avidemux creates an outputfile with multiple IDs (e.g. one for each input video), so that playback of the output file is possible.

Negative side-effect is of course, that such an output file could cause problems if the user would like to post-process such an output file, e.g. also in avidemux and again with copy-mode.

I understand that this sounds much easier than it is in real life. Also there's an easy and 'acceptable' workaround by switching from copy-mode to re-encoding mode. And finally, it seems that this problem doesn't occur very often, i.e. usecases are limited.

[eumagga0x2a]

Therefore the usecase regarding avidemux would be:

• User adds multiple videos to avidemux (all have ID=0)
• User saves them in copy-mode
• Avidemux creates an outputfile with multiple IDs (e.g. one for each input video), so that playback of the output file is possible.

There is actually no need for multiple IDs as long as in-band SPS is used and most video players can handle AVCC (ISO, mp4) type H.264 streams with in-band parameter sets (which take precedence over the global one). But without in-band parameter sets, a different SPS needs to be referenced from a different PPS which in turn is referenced by slice headers via PPS id. It is not necessary to re-encode macroblocks, but both the SPS and PPS in the global header and all following slice headers need to be re-created, which is black magic and thus usually the job of the encoder.

All SPS checks in Avidemux would need to parse the PPS first, then partially decode slice header to know whether this SPS applies to this slice or not. A whole lot of extra complexity.