DVB TS h264 1080i streams incorrectly decoded as r601 when it should be r709

[radorn]

Same MEGA folder. ~7MB, 10 complete GOPs.
Sounds OK?

[eumagga0x2a]

Perfect, thank you very much.

[radorn]

So, going back to the original topic, is there any workaround I can try to correct the color?

[eumagga0x2a]

You haven't provided a project script which would allow to judge what you actually do with the stream (the configured filter chain, the encoder and its settings), but you should be able to specify color properties manually when encoding to HEVC using x265. While x264 supports color properties, Avidemux doesn't provide means to pass this configuration to the encoder library.

When muxing into a MKV (which doesn't work really well in copy mode), you can specify color properties at the container level. A video player (not Avidemux for the purpose of display) might respect these settings.

[radorn]

The encodes I'm tryingo to produce are mainly intended for that that same USB-enabled tuner, for size reduction and removing ads and so on (GOPs are 1+ seconds long, so it's not good when they don't fully align with these). This device doesn't seem to respect any matroska off-band signaling, but rather it seems only interested in what the audio and video streams might have to say. I doesn't even obey MKV aspect ratio, so that doesn't seem like a promising route. It's a miracle it supports matroska at all.
The best video codec that this thing supports is h264, so that's what I'll be using, but while Avidemux's x264 encoder does have a tab called "Output2" which seems to have the relevant options for the task, these are all grayed out, both in basic or advanced configuration modes. I can't seem to find a way to enable those.

It seems to work with x265. Now the decoded video has the same colors as the TS. But that tuner won't play h265 video.

[eumagga0x2a]

while Avidemux's x264 encoder does have a tab called "Output2" which seems to have the relevant options for the task, these are all grayed out, both in basic or advanced configuration modes. I can't seem to find a way to enable those.

This functionality is not implemented in the backend yet.

Regarding display, D3D9 we use for hw accelerated display on Windows AFAIK provides no control over colorspace.

[radorn]

I guess this is no longer related to Avidemux anymore, but, Do you know any way I can insert that kind of color information into the h264 stream after the encoding is done?
I know I can set them in the matroska container, but, as I said, that'll be ignored by the DVB tuner I intend these encodes for.

[eumagga0x2a]

Do you know any way I can insert that kind of color information into the h264 stream after the encoding is done?

For formats with out of band extradata, this would require at least generating extradata for the given configuration with this information added – a hardly practicable way, which would require a lot of skilled coding. If this color information makes x264 actually generate a different compressed stream instead of being just a hint stored in the SPS, the task would become totally unfeasible.

[eumagga0x2a]

By the way, the issue is that you can't save the entire MKV you provided as MP4 due to broken DTS for the first few frames.

Should be fixed by [demuxers/Matroska] Take early B-frames into account when enforcing fixed frame rate, thank you again for the sample.

I'll try to add support for color information to the backend of the x264 encoder plugin. Might take a couple of days.

[radorn]

For formats with out of band extradata, this would require at least generating extradata for the given configuration with this information added – a hardly practicable way, which would require a lot of skilled coding. If this color information makes x264 actually generate a different compressed stream instead of being just a hint stored in the SPS, the task would become totally unfeasible.

I understand. I was just figuring that, perhaps, it could be just some flag in the bitstream or something "simple" like that and, maybe, there was a tool for it already available which you might know about. If that's not the case, it's OK.

thank you again for the sample.

You're welcome.

And now that you mention B-frames... I have been wondering for some time now:
Is it safe to cut a GOP leaving one of it's B-frames hanging at the end, be it before the end of the file or in front of a different frame than that which followed it originally? Or should I always take care to always leave a GOP "closed" by at least one of it's original back-referencing-only P-frames, so that all B-frames left in will have their intended forward referenced frames preserved in the editted stream?
Do the trailing B-frames of one GOP ever forward-reference the starting I-frame or K-frame of the next GOP or are all GOPs entirely independent and selfcontained?

I ask this because I've been cutting up videos in copy mode for some time now, and I've recently noticed some, to me, unexpected problems around cut points in certain files, and I'm wondering if, perhaps, my method was wrong meaning I might have left behind a bunch of mangled cutpoints in previously editted files.
B-frames are the only explanation I can think about right now.

I'll try to add support for color information to the backend of the x264 encoder plugin. Might take a couple of days.

Thank you very much. Apreciated.

[eumagga0x2a]

Done, with two regressions immediately spotted and fixed by followup commits. May still be buggy though.

[eumagga0x2a]

Is it safe to cut a GOP leaving one of it's B-frames hanging at the end, be it before the end of the file or in front of a different frame than that which followed it originally?

You can set the end of selection for the purpose of saving it when in copy mode to any frame, Avidemux will cut the stream short so no frame is left dangling. Setting the B marker to a B-frame will exclude that frame and all later frames in display order from being included into the output, which probably cuts away the P-frame this B-frame depends on and likely all B-frames before until the previous P-frame (or keyframe).

Or should I always take care to always leave a GOP "closed" by at least one of it's original back-referencing-only P-frames, so that all B-frames left in will have their intended forward referenced frames preserved in the editted stream?

If I am not mistaken, one doesn't need to take care of that, but placing the end of selection on a keyframe reduces the stretch at the end of the video missing in the output. All forward references in display order obviously reference only past frames in stream order.

Do the trailing B-frames of one GOP ever forward-reference the starting I-frame or K-frame of the next GOP or are all GOPs entirely independent and selfcontained?

In closed-GOP streams, GOPs are self-contained, but you probably won't find public digital broadcasts which still use closed GOPs as they are detrimental to quality in the presence of bandwidth constraints. Keyframes in usual open-GOP streams are merely seek points, they guarantee that you can start decoding the stream from this point and get no incomplete pictures. But all the usual dependencies continue across such seek points. Defining a GOP as a sequence of slices from one IDR slice to the next one, many such streams never see a GOP end, or they may go on for minutes, hours or days.

Such streams are fundamentally incompatible with cuts within a stream (i.e. with deletions or insertions), but many decoders handle strictly speaking invalid streams with such cuts quite graciously with libavcodec-based players and editors like Avidemux being really bad here. Avidemux performs some checks to ensure that the worst problems like video stopping for many seconds don't happen and warns the user otherwise.

[radorn]

Sorry for the delay.

Soon I'll be trying a new compile with the color-setting settings you added to the x264 encoder and tell you how that went. Thanks again.

I'm also doing some tests with avidemux and other programs to get a better grasp of how all these things work together, and, also, to see what are the precise limits of that settopbox I use. The manual seems to indicate high profile, level 4, and that's it. Nothing about other h264-related details or container settings. Not sure if that includes levels 4.1 and 4.2 either.

Anyway, VirtualDub2's built-in x264 encoder's commandline arguments list reveals this:

Video Usability Info (Annex E):
The VUI settings are not used by the encoder but are merely suggestions to
the playback equipment. See doc/vui.txt for details. Use at your own risk.

That's the header for the section with all the settings for color primaries, transfer functions and so on. So it seems these things are implemented as flags in the h264 bitstream but they don't affect the encoded video after all. Not that this solves the problem of adding that information to, or modifying it in, an already encoded h264 stream, of course.

Thanks for the explanation on GOPs. I've seen "open GOP" and "closed GOP" written many times but never took the time to find out what it meant.

It'd be great if there were algorightms in place for stream-copy edition where only the necessary missing parts of GOPs were re-encoded so to have a proper lossless edit in copy mode no matter where you cut the stream. I'm just daydreaming of course, and I realize that the ammount of research and engineering this would require (and replicated for every coded under the sun) for something so niche would be insane and hardly justifiable.
Being practical, if the edition needs to be fancier, just re-encode.

Most channels I usually record are still SD MPEG2 DVB-1, and these lend themselves better to being just chopped up in copy mode, especially if they do fadeouts and fadeins when going in and coming ouf of commercials. And I don't have to worry about color primaries with these. But other times they just jump from the running movie to the network branding transition from one frame to the next, and I started noticing funny stuff with these cut points sometimes. I guessed that the problem could be related to the preceeding GOP ending with a B-frame that's now missing it's forward reference.

With those things in mind, and, more important, the rather big size of these edits, I started finding it necessary to re-encode.

------

Well, this is getting long. I'll update you with my results when I get around trying your fix. And thanks again.

[radorn]

I meant to test your update, but the last available compile turns out to be older. Unfortunately I wasted another 30MB to find that out. I need to be more careful.

Anyway. After quite a bit of experimentation with avidemux, Vdub, and others, I've come to the unfortunate conclusion that this stupid DVB tuner doesn't even honor VUI information. So all the work in enabling it won't benefit my use of this junk settopbox after all. I'm really sorry.
The only thing that would do the trick in this case of mine, would be to implement actual color conversion from bt709 to bt601. Of course, I understand this is way more involved, so don't worry.

Anyway, this device is so dumb that even h264 SAR (why don't the call it Pixel Aspect Ratio like everyone else?) isn't properly interpreted. For starters the player only really supports 2 projection modes: 4:3 and 16:9, and the actual resolution of the material doesn't seem to count for anything. Somehow the player decides whether a particular video file should be projected one of those two ways, and just stretches whatever resolution it finds and fills that rectangle with it. Limited, but workable. But what's really bonkers is how it just does whatever it wants with the VUI SAR info. A SAR of 1:1 always results in widescreen projection, and the actual resolution has no bearing on the result. Other SAR values somehow result in wide or normal projections but the rules that decide this definitely don't conform to real AR calculations, mpr do they seem to follow any apparent logic. But, in the end, at least I now have a system for it:  1:1 SAR results in 16:9 widescreen and 1:2 SAR spits 4:3. Nonsensical, but works every time.
Cheap made-in-prc junk.

So, in the end, for regular DVB-1 channels that broadcast SD MPEG2 video in bt601 color, avidemux works perfectly fine for me. But for recordings of DVB-2 channels with HD AVC/h264 video in bt709 color, I would need a way to actually convert the colorspace. I downscale them to SD anyway, as I'm still watching on a CRT.

[eumagga0x2a]

I meant to test your update, but the last available compile turns out to be older.

Refresh of the nightlies requested, it will happen when it happens.

But for recordings of DVB-2 channels with HD AVC/h264 video in bt709 color, I would need a way to actually convert the colorspace.

Honestly, I never perceived the difference (I use my computer monitor for all purposes). The one which is sorely missing, is a proper conversion bt2020 --> bt709, but I was never bothered by bt601/bt709 mismatch as long as color range was correct. Better support for colorspaces will hopefully come some day.

I downscale them to SD anyway, as I'm still watching on a CRT.

Over the time, the cost for wasted electricity caused by the CRT might have exceeded the price of a new LCD... Even decade old TV sets allow to plug in a flash drive and support a wide range of formats. However, the longevity of that CRT is amazing.