Avidemux Forum

Avidemux => Main version 2.6 => Topic started by: gamez on October 31, 2020, 10:27:09 PM

Title: 2.7.6 crashes reproducibly on its own py script: Cannot set up encoder
Post by: gamez on October 31, 2020, 10:27:09 PM
Hi

I am transcoding and cutting into pieces a TV recording from MPEG2 TS to MP4 format:

#PY  <- Needed to identify #
#--automatically built--

adm = Avidemux()
if not adm.loadVideo("Z:/TVRip/20180113 1911 - ProSieben - Storm Hunters.ts"):
    raise("Cannot load Z:/TVRip/20180113 1911 - ProSieben - Storm Hunters.ts")
adm.clearSegments()
adm.addSegment(0, 173016240, 918760016)
adm.addSegment(0, 1681096192, 783000064)
adm.addSegment(0, 3069656320, 1342759680)
adm.addSegment(0, 4939136512, 1209920000)
adm.addSegment(0, 6826176512, 574719552)
adm.markerA = 0
adm.markerB = 4829159312
adm.setPostProc(3, 3, 0)
adm.videoCodec("x264", "useAdvancedConfiguration=True", "general.params=2PASSBITRATE=3376", "general.threads=0", "general.preset=ultrafast", "general.tuning=none", "general.profile=baseline", "general.fast_decode=False", "general.zero_latency=False"
, "general.fast_first_pass=True", "general.blueray_compatibility=False", "general.fake_interlaced=False", "level=-1", "vui.sar_height=1", "vui.sar_width=1", "MaxRefFrames=3", "MinIdr=25", "MaxIdr=250"
, "i_scenecut_threshold=40", "intra_refresh=False", "MaxBFrame=3", "i_bframe_adaptive=1", "i_bframe_bias=0", "i_bframe_pyramid=2", "b_deblocking_filter=True", "i_deblocking_filter_alphac0=0", "i_deblocking_filter_beta=0"
, "cabac=True", "interlaced=False", "constrained_intra=False", "tff=True", "fake_interlaced=False", "analyze.b_8x8=True", "analyze.b_i4x4=True", "analyze.b_i8x8=True", "analyze.b_p8x8=True", "analyze.b_p16x16=False"
, "analyze.b_b16x16=False", "analyze.weighted_pred=2", "analyze.weighted_bipred=True", "analyze.direct_mv_pred=1", "analyze.chroma_offset=0", "analyze.me_method=1", "analyze.me_range=16", "analyze.mv_range=-1"
, "analyze.mv_range_thread=-1", "analyze.subpel_refine=7", "analyze.chroma_me=True", "analyze.mixed_references=True", "analyze.trellis=1", "analyze.psy_rd=1.000000", "analyze.psy_trellis=0.000000", "analyze.fast_pskip=True"
, "analyze.dct_decimate=True", "analyze.noise_reduction=0", "analyze.psy=True", "analyze.intra_luma=11", "analyze.inter_luma=21", "ratecontrol.rc_method=0", "ratecontrol.qp_constant=0", "ratecontrol.qp_min=10"
, "ratecontrol.qp_max=51", "ratecontrol.qp_step=4", "ratecontrol.bitrate=0", "ratecontrol.rate_tolerance=1.000000", "ratecontrol.vbv_max_bitrate=0", "ratecontrol.vbv_buffer_size=0", "ratecontrol.vbv_buffer_init=1"
, "ratecontrol.ip_factor=1.400000", "ratecontrol.pb_factor=1.300000", "ratecontrol.aq_mode=1", "ratecontrol.aq_strength=1.000000", "ratecontrol.mb_tree=True", "ratecontrol.lookahead=40")
adm.addVideoFilter("swscale", "width=720", "height=404", "algo=2", "sourceAR=0", "targetAR=0", "lockAR=False", "roundup=False")
adm.audioClearTracks()
adm.setSourceTrackLanguage(0,"deu")
adm.setSourceTrackLanguage(1,"deu")
adm.audioAddTrack(0)
adm.audioCodec(0, "copy", "bitrate=192", "mode=0");
adm.audioSetDrc(0, 0)
adm.audioSetShift(0, 0, 0)
adm.setContainer("MP4V2", "optimize=0", "add_itunes_metadata=0")

This py script was saved before pushing the "save" button and was processed OK.

However when I close avidemux, reopen the script (which opens and displays he cut TS OK) and try to save it again, avidemux crashes with a "Cannot set up encoder. The configuration supplied to the encoder may be incompatible or the encoder may depend on features unavailable on this system." popup message. The _cli pretty much behaves the same way.


Shouldn't avidemux be able to re-execute its own scripts?

g.
Title: Re: 2.7.6 crashes reproducibly on its own py script: Cannot set up encoder
Post by: eumagga0x2a on November 01, 2020, 09:25:47 PM
A few notes to the setup:

Quote from: gamez on October 31, 2020, 10:27:09 PMadm.setPostProc(3, 3, 0)

Down with post-processing. At best (video is decoded in hardware), it does nothing.

Quote from: gamez on October 31, 2020, 10:27:09 PMadm.addVideoFilter("swscale", "width=720", "height=404", "algo=2", "sourceAR=0", "targetAR=0", "lockAR=False", "roundup=False")

The source is most likely interlaced. If it is interlaced, you should insert a deiterlacing filter as the very first transformation filter in the chain.

Quote from: gamez on October 31, 2020, 10:27:09 PMadm.audioCodec(0, "copy", "bitrate=192", "mode=0");

I wonder how do you manage to get this mixup (the dummy codec "copy" doesn't allow any parameters).

Quote from: gamez on October 31, 2020, 10:27:09 PMadm.setContainer("MP4V2", "optimize=0", "add_itunes_metadata=0")

The MP4v2 muxer is deprecated, please use MP4.

Quote from: gamez on October 31, 2020, 10:27:09 PMHowever when I close avidemux, reopen the script (which opens and displays he cut TS OK) and try to save it again, avidemux crashes with a "Cannot set up encoder. The configuration supplied to the encoder may be incompatible or the encoder may depend on features unavailable on this system." popup message.

You cite a pretty clear formulated error message, do you really experience a subsequent crash? If you do, please provide info about the crash, there might be some in %localappdata%\avidemux\admlog.txt (the log file will be overwritten when Avidemux is restarted).

Quote from: gamez on October 31, 2020, 10:27:09 PMShouldn't avidemux be able to re-execute its own scripts?

It has successfully executed the script. It doesn't verify that the configuration passed to the encoder it to encoder liking, however. This is beyond the scope of the internal script writer.

In any case, please use a more recent build, The last release (2.7.6) has got long in the tooth.
Title: Re: 2.7.6 crashes reproducibly on its own py script: Cannot set up encoder
Post by: gamez on November 03, 2020, 07:33:38 AM
Hi

I am very confused by your reply. The script was as a whole created and generated by avidemux itself without any manual changes from my side. This should clarify your questions related to setPostProc and audioCodec.

Regarding setContainer: I had also tried all other available containers with the exact same result. avidemux crashes already much earlier, when trying to start and setup the encoder. The conatiner is not the issue in my belief. I also tried many other codecs, again with the exact same result.

QuoteThe MP4v2 muxer is deprecated, please use MP4.
I thought it was the other way around... Never mind, the "old" MP4 produces the exact same error.

avidemux doesn't crash. It gives an error and says that outputting the file has failed.

BTW, what is very strange: When I load the script, the very first frame from the .ts is shown in the display, although the first segment starts with a PTS of 2 min 53 sec into the file. Only when I start navigating, the segment's first frame is shown at position 0 (actually not quite zero, but at 256 ms). Also, the "Cannot set up encoder" is followed by "Error seeking to 256 ms".

Actually I think there is a problem in avidemux when he tries to find the start of the first PTS in the segment, and using that as the new starting point of its processing.

g.
Title: Re: 2.7.6 crashes reproducibly on its own py script: Cannot set up encoder
Post by: eumagga0x2a on November 03, 2020, 03:31:31 PM
Quote from: gamez on November 03, 2020, 07:33:38 AMThe script was as a whole created and generated by avidemux itself without any manual changes from my side. This should clarify your questions related to setPostProc and audioCodec.

I pointed out some things in the script which raised questions. They are not necessarily knife-sharp focussed on addressing the failure to setup encoder.

I recommended you to disable post-processing in Avidemux preferences. The scripted audioCodec configuration is invalid, I need to check how it is possible that Avidemux creats it this way.

Quote from: gamez on November 03, 2020, 07:33:38 AMRegarding setContainer: I had also tried all other available containers with the exact same result.

I didn't mean that. Just please don't use MP4v2 if you want to save to mp4, use MP4. Of course, this is not related to the failure.

Quote from: gamez on November 03, 2020, 07:33:38 AMI thought it was the other way around...

Exactly. libmp4v2-based "MP4v2" is old and stale (it is not present in VC++ builds anyway), the other one is very much current.

Quote from: gamez on November 03, 2020, 07:33:38 AMActually I think there is a problem in avidemux when he tries to find the start of the first PTS in the segment, and using that as the new starting point of its processing.

This is another, unrelated problem, I think.
Title: Re: 2.7.6 crashes reproducibly on its own py script: Cannot set up encoder
Post by: gamez on November 03, 2020, 04:53:40 PM
Hi

The hickup in the audioCodec config string seems to be related to switching from a compressor setting (like AC3 Aften) to the copy setting, and/or in-/excluding source audio tracks into the target (actually my .ts recording has one mp2 and one ac3 track).

I strongly believe that there is a problem with carrying over the selected range of frames of the segments from the interactive editing to the py script, and/or using those directly from a reloaded py script without loading the base file (.ts) in the first place. There is some de-synchronization going on that I will try to isolate further in my tests. Could also be related to the type of frame (I, B or P) that is present at the segment borders since in case of an MPEG-2 transport stream the full frame(s) needed to construct an image inside the selected segment(s) might lie outside of it.

I will let you know what I find.

g.
Title: Re: 2.7.6 crashes reproducibly on its own py script: Cannot set up encoder
Post by: eumagga0x2a on November 03, 2020, 05:42:07 PM
Please just leave out all potentially problematic parts for now and add the deinterlacer (and use the latest nightly, of course):

#PY  <- Needed to identify #
#--automatically built--

adm = Avidemux()
if not adm.loadVideo("Z:/TVRip/20180113 1911 - ProSieben - Storm Hunters.ts"):
    raise("Cannot load Z:/TVRip/20180113 1911 - ProSieben - Storm Hunters.ts")
adm.videoCodec("x264", "useAdvancedConfiguration=True", "general.params=2PASSBITRATE=3376", "general.threads=0", "general.preset=ultrafast", "general.tuning=none", "general.profile=baseline", "general.fast_decode=False", "general.zero_latency=False"
, "general.fast_first_pass=True", "general.blueray_compatibility=False", "general.fake_interlaced=False", "level=-1", "vui.sar_height=1", "vui.sar_width=1", "MaxRefFrames=3", "MinIdr=25", "MaxIdr=250"
, "i_scenecut_threshold=40", "intra_refresh=False", "MaxBFrame=3", "i_bframe_adaptive=1", "i_bframe_bias=0", "i_bframe_pyramid=2", "b_deblocking_filter=True", "i_deblocking_filter_alphac0=0", "i_deblocking_filter_beta=0"
, "cabac=True", "interlaced=False", "constrained_intra=False", "tff=True", "fake_interlaced=False", "analyze.b_8x8=True", "analyze.b_i4x4=True", "analyze.b_i8x8=True", "analyze.b_p8x8=True", "analyze.b_p16x16=False"
, "analyze.b_b16x16=False", "analyze.weighted_pred=2", "analyze.weighted_bipred=True", "analyze.direct_mv_pred=1", "analyze.chroma_offset=0", "analyze.me_method=1", "analyze.me_range=16", "analyze.mv_range=-1"
, "analyze.mv_range_thread=-1", "analyze.subpel_refine=7", "analyze.chroma_me=True", "analyze.mixed_references=True", "analyze.trellis=1", "analyze.psy_rd=1.000000", "analyze.psy_trellis=0.000000", "analyze.fast_pskip=True"
, "analyze.dct_decimate=True", "analyze.noise_reduction=0", "analyze.psy=True", "analyze.intra_luma=11", "analyze.inter_luma=21", "ratecontrol.rc_method=0", "ratecontrol.qp_constant=0", "ratecontrol.qp_min=10"
, "ratecontrol.qp_max=51", "ratecontrol.qp_step=4", "ratecontrol.bitrate=0", "ratecontrol.rate_tolerance=1.000000", "ratecontrol.vbv_max_bitrate=0", "ratecontrol.vbv_buffer_size=0", "ratecontrol.vbv_buffer_init=1"
, "ratecontrol.ip_factor=1.400000", "ratecontrol.pb_factor=1.300000", "ratecontrol.aq_mode=1", "ratecontrol.aq_strength=1.000000", "ratecontrol.mb_tree=True", "ratecontrol.lookahead=40")
adm.addVideoFilter("yadif", "mode=1", "parity=0", "deint=0")
adm.setContainer("MP4", "muxerType=0", "optimize=1", "forceAspectRatio=True", "aspectRatio=1", "rotation=0", "clockfreq=0")

If x264 still doesn't like the configuration, we should check deeper (libx264 used by my Avidemux build accepts it).
Title: Re: 2.7.6 crashes reproducibly on its own py script: Cannot set up encoder
Post by: gamez on November 03, 2020, 07:26:15 PM
The problem is related to segments and/or markers.
Running the encoder with the whole .ts file never did give any problems, neither in interactive nor in batch mode.
Title: Re: 2.7.6 crashes reproducibly on its own py script: Cannot set up encoder
Post by: eumagga0x2a on November 03, 2020, 07:45:30 PM
Please provide the log.
Title: Re: 2.7.6 crashes reproducibly on its own py script: Cannot set up encoder
Post by: gamez on November 03, 2020, 08:24:12 PM
So it turns out that, occasionally, the number saved to the py script to indicate the start of the segment is wrong.

For example, when I cut the .ts at the I-frame at 04:54.496 this is saved as 294496244. This can be explained by 4 mins and 54 secs and 496 ms being equal to 294496 ms, followed by the number 244 I cannot explain. An encoder run (in batch mode with --load py and with this starting position succeeds.

The next frame at position 04:54.536 (in a 25 fps .ts video, every frame "occupies" 40 ms) is a B-frame and is saved as 294536244. An encoder run also works OK.

But then, the next frame at 04:54.576 is again a B-frame and for some reason saved as 294576256, and the encoder crashes immediately. When I change it to 294576244, it succeeds.

Again, the next frame at 04:54.616 is a P-frame and is wrongly saved as 294616256, and the encoder crashes. After correction to 294616244, it works.

I don't know what the trailing 244 (or 256) stands for, but as least I have found the reason.

g.
Title: Re: 2.7.6 crashes reproducibly on its own py script: Cannot set up encoder
Post by: eumagga0x2a on November 03, 2020, 09:34:06 PM
Quote from: gamez on November 03, 2020, 08:24:12 PMFor example, when I cut the .ts at the I-frame at 04:54.496 this is saved as 294496244. This can be explained by 4 mins and 54 secs and 496 ms being equal to 294496 ms, followed by the number 244 I cannot explain.

Internally, Avidemux keeps track of time in microseconds (µs).

Quote from: gamez on November 03, 2020, 08:24:12 PMBut then, the next frame at 04:54.576 is again a B-frame and for some reason saved as 294576256, and the encoder crashes immediately.

Might be some rounding error. Please provide the log from the latest nightly after deleting *.idx2 so that I can assess the situation. A failure to setup encoder is not a crash.
Title: Re: 2.7.6 crashes reproducibly on its own py script: Cannot set up encoder
Post by: gamez on November 04, 2020, 08:39:42 AM
QuotePlease provide the log

I would love to, if only this forum would allow me to embed it into my post, attach to it as zip, or link to it on pastebin. But all of that is detected as spam and forbidden. I guess I'm out of luck and options.

BTW, besides the bugs with the audioCodec conf and the glitch in recording correct microseconds, I'm convinced there's a bug in having addSegment and markerA/B play nicely together with each other whenever a segment doesn't start at file position 0. In such cases the script runs differently depending on whether you specify adm.markerA = 0 and adm.markerB = <length_of_segment> or leave them out. Normally the result should be the same but it is not. And as soon as you stack several segments together, the results seem to become fully unpredictable. I am suspecting that the markers sometimes refers back to the initial .ts file from where the segments stem from.

I will continue my investigations and post what I find (if the forum allows me to).

g.
Title: Re: 2.7.6 crashes reproducibly on its own py script: Cannot set up encoder
Post by: eumagga0x2a on November 04, 2020, 09:59:29 AM
If CleanTalk misbehaves again and blocks totally legitimate posts as spam, it may help trying to post from a private browser window, else please don't hesitate to provide the info per private message.

It is definitely possible to attach files with filename extension .txt like happened here (https://avidemux.org/smif/index.php/topic,19304.0.html).

addSegment() expects time in reference (in source video) as the second and the third argument, their values are specific to the particular video. The markers A and B are a completely different story.
Title: Re: 2.7.6 crashes reproducibly on its own py script: Cannot set up encoder
Post by: gamez on November 04, 2020, 05:41:16 PM
Hi

tx for the suggestion, so I sent you a PM.

Besides, in your view, should putting adm.markerA = 0 and adm.markerB = <sum_of_length_of_segment(s)> yield the same result as leaving them both out (what is what I preferably do)? In all my logic the answer should be yes, but in fact, it is no.

In my understanding, markers delimit the portion of the file that you want to encode. The problem here is, what means "file": the base (.ts) file or the concatenated segments?

In my tests, for a single segment, it's still quite OK, but for more than one concatenated segment with gaps in between, the encoding goes a little berserk and tends to continue after the end of the first segment instead of "jumping" to the second one, sometimes even until the very end of the base .ts file. So yes, there is interdependance between markers and segments.

g.
Title: Re: 2.7.6 crashes reproducibly on its own py script: Cannot set up encoder
Post by: eumagga0x2a on November 04, 2020, 07:41:15 PM
[setup] 22:22:00-665 Creating x264 encoder
[setup] 22:22:00-665 x264, setup ok

Encoder setup is not an issue, the reason for failure is

[TimeToFrame] 22:22:00-650 We reached frame 7362 with a PTS of 294616244 when looking for PTS 294576256
[getFrameNumFromPtsOrBefore] 22:22:00-650 Best candidate for time 00:04:54,576 in reference is frame 7361
[seektoTime] 22:22:00-650 Seeking to frame 7356 at 294496 ms
[DecodePictureUpToIntra] 22:22:00-650 Decoding up to intra frame 7356, ref: 0
[edCache] Flush
[seektoTime] 22:22:00-665 Image not found, looked for 294576256 us, got pts=294616244 us

the specified start of the segment in reference which doesn't match a frame.

# adm.setPostProc(3, 3, 0)
This is not what I meant, I suggested to disable post-processing in Avidemux preferences alltogether, equivalent to

adm.setPostProc(0, 0, 0)

*************************
  Avidemux v2.7.7 (200714_c5a3dd2fb4b) .
*************************
 http://www.avidemux.org
 Code      : Mean, JSC, Grant Pedersen,eumagga0x2a
 GFX       : Nestor Di, nestordi@augcyl.org
 Design    : Jakub Misak
 FreeBSD   : Anish Mistry, amistry@am-productions.biz
 Audio     : Mihail Zenkov
 Mac OS X  : Kuisathaverat, Harry van der Wolf
 Win32     : Grant Pedersen
 
Compiler: GCC 5.4.0
Build Target: Microsoft Windows (x86)
User Interface: CLI
Operating System: Microsoft Windows (6.2.9200; 32-bit)
Time: Tue Nov 03 23:22:00 2020
 
0: C:\Program Files\Avidemux 2.7 - 32 bits\avidemux_cli.exe

I really should have asked first. The legacy compatibility branch, used to generate 32-bit builds, has not been updated since release, I don't know whether I'll be able to backport the truckload of commits from git master for a final 32-bit 2.7.7 (which will be the anticipated end of support for Windows XP and 32-bit in general). I can only recommend to switch to a 64-bit operating system.
Title: Re: 2.7.6 crashes reproducibly on its own py script: Cannot set up encoder
Post by: eumagga0x2a on November 04, 2020, 07:54:44 PM
Quote from: gamez on November 04, 2020, 05:41:16 PMBesides, in your view, should putting adm.markerA = 0 and adm.markerB = <sum_of_length_of_segment(s)> yield the same result as leaving them both out (what is what I preferably do)? In all my logic the answer should be yes, but in fact, it is no.

In my logic, the answer is "yes" too.

Quote from: gamez on November 04, 2020, 05:41:16 PMIn my understanding, markers delimit the portion of the file that you want to encode. The problem here is, what means "file": the base (.ts) file or the concatenated segments?

Markers are in linear time (sum of segment durations), the number of files is irrelevant. Segments are in reference time.

Quote from: gamez on November 04, 2020, 05:41:16 PMIn my tests, for a single segment, it's still quite OK, but for more than one concatenated segment with gaps in between, the encoding goes a little berserk and tends to continue after the end of the first segment instead of "jumping" to the second one, sometimes even until the very end of the base .ts file.

Specifying arbitrary values for segment start / end may cause unexpected behaviour, right. Doing this may break segment switching. Values, created by Avidemux GUI, should work if the video is free of defects (except for bugs – in this matter it is fatal that you cannot try a current build as you run a 32-bit Windows).
Title: Re: 2.7.6 crashes reproducibly on its own py script: Cannot set up encoder
Post by: gamez on November 05, 2020, 08:18:47 AM
OK, let me retry on 64 bit OS
g.
Title: Re: 2.7.6 crashes reproducibly on its own py script: Cannot set up encoder
Post by: eumagga0x2a on November 05, 2020, 08:47:47 AM
Quote from: gamez on November 05, 2020, 08:18:47 AMOK, let me retry on 64 bit OS

...with the latest 64-bit Avidemux nightly, please.
Title: Re: 2.7.6 crashes reproducibly on its own py script: Cannot set up encoder
Post by: gamez on November 05, 2020, 01:40:16 PM
Quotelatest 64-bit Avidemux nightly
Yes sure - I figure it's the 201102_ffa9a6d4b29?

So the good news is that
a) it now correctly saves the border frames' µs as .........244 and never .........256
b) in consequence has no more seeking problems.

There's still a (little) question for the markers in conjunction with stacked segments.

In my tests I was doing
adm.addSegment(0, 173056244, 918760000)
adm.addSegment(0, 1681216244, 790080000)
adm.addSegment(0, 3076816244, 1335600000)
adm.addSegment(0, 4939096244, 1209960000)
adm.addSegment(0, 6826136244, 574720000)
with and without
adm.markerA = 0
adm.markerB = 4829120000
Note that markerB sits at the end of the sum of the segments.

When I set the markers, the resulting file size is 2.390.157.657, when I leave them out it's  2.390.161.909.

Where does the 4K difference come from?

g.
Title: Re: 2.7.6 crashes reproducibly on its own py script: Cannot set up encoder
Post by: eumagga0x2a on November 05, 2020, 09:53:14 PM
Quote from: gamez on November 05, 2020, 01:40:16 PMa) it now correctly saves the border frames' µs as .........244 and never .........256

I wish I could attribute this to a particular code change.

Quote from: gamez on November 05, 2020, 01:40:16 PMWhen I set the markers, the resulting file size is 2.390.157.657, when I leave them out it's  2.390.161.909.

How many frames are written to the output file in the first vs. the second case? I wonder what does

print("marker A = " + str(adm.markerA) + " marker B = " + str(adm.markerB) + "\n")
report to the log when added after all the addSegment() calls.

I assume that you have not forgotten to clearSegments() prior to creating your custom segment layout using addSegment().