Have problem with CLI and Scripts on Windows build 2.6 versions

[SeventyPlus]

I've seen mean made new revision available, can you test that one? most likely dealing with your issue?
2.60 r7996 http://avidemux.org/nightly/

Allright, I did my test  of the DOS command, with avidemux_cli.exe (CLI avidemux from nightly 2.60 r7996 win32) in the command-line and Python .py script as the avidemux argument.

The DOS command line is as follows:

Code Select Expand


"D:\users\hv\AVIDEMUX\avidemux_r7996_win32\Release\avidemux_cli.exe" --runpy "C:/My Programs and Doc/MyMpegJoiner/TEMPLATES/TestScript_SimpleJoin_BetaVersion.py" --quit >"C:\My Programs and Doc\MyMpegJoiner\LOG\TestScript_SimpleJoin_BetaVersion.log" 2<&1


The script file TestScript_SimpleJoin_BetaVersion.py contains following script:

Code Select Expand


#PY  <- Needed to identify#
#--automatically built--
#--Project: C:/My Programs and Doc/MyMpegJoiner/TEMPLATES/TestScript_SimpleJoin_BetaVersion.py

adm=Avidemux()
targetFile="E:/PANASONIC_CAPTURES/STEP_3_JOIN_AVIDEMUX_AVI/VARIA1/OUTPUT_BY_PY.MPG"
#** Video **
# 03 videos source
adm.loadVideo("E:/PANASONIC_CAPTURES/STEP_3_JOIN_AVIDEMUX_AVI/VARIA1/2000-01-01_00.MPG")
adm.appendVideo("E:/PANASONIC_CAPTURES/STEP_3_JOIN_AVIDEMUX_AVI/VARIA1/2000-01-01_01.MPG")
adm.appendVideo("E:/PANASONIC_CAPTURES/STEP_3_JOIN_AVIDEMUX_AVI/VARIA1/2000-01-01_02.MPG")
#** Postproc **
adm.setPostProc(0,0,0)

#** Video Codec conf **
adm.videoCodec("Copy")

#** Filters **

#** Audio **
adm.audioClearTracks()

#** Track 0 **
adm.audioAddTrack(0)
adm.audioCodec(0,"copy");
adm.audioSetPal2Film(0,0)
adm.audioSetFilm2Pal(0,0)
adm.audioSetNormalize(0,0,0)

#** Muxer **
adm.setContainer("ffPS","muxingType=2","acceptNonCompliant=True","muxRatekBits=11000","videoRatekBits=9800","bufferSizekBytes=224")
adm.save(targetFile)
#End of script


The log file with redirected StandardOut and redirected ErrorOut of the test with avidemux_cli.exe 2.60 r7996 is in attachment.
(I made a reduction: deleted similar error messages because the original log file is really hugh)

These are the results:

(1) There is progress in so far that now the 'appends' of the other source files start working (in the previous builds only the 'load' of the first source seemed to work).

But what strikes me: at every append of a source file, the program seems to do a rescan of ALL similar .mpg source files as if the program is willing to do an automatic append.
So : if for example a directory contains 100 .mpg source files and I want to join (Load+append) these 100 .mpeg source files, than the program will do 100*100 rescans (=10 000). Does not seem performant (and will flood a log file or in my case a streamreader who will be "listening" for progress information).
Reminder: this rescan does not happen with the v2.5.6 version. In the log file of a similar run with v2.5.6 avidemux_cli.exe, you will find a message like "There was several files, but dont append was forced" and no more scans are done when each source file is appended.

(2) After the 'load'+'append' messages in the log file followed a hugh portion (more than 2 megabytes, now reduced) of error overflow messages (as a result of these errors, no output is generated): this is certainly something to work on.

[mean]

The buffer underflow warning are not really a problem
I'm more concerned about these ones

Code Select Expand


[AudioStream] Warning skew in dts =-55608000,
[AudioStream] Warning skew lastDts= 00:00:55,648 
[AudioStream] Warning skew newDts= 00:00:00,040   


Could you provide a couple of files which, when joined produce these messages ?
They dont need to be be long

[mean]

If the files are too large, does these skew messages appear when you use the Qt4 version ?

[SeventyPlus]

If the files are too large, does these skew messages appear when you use the Qt4 version ?

First of all: thank you very much for spending your time on my problem case.

As to Qt4 version : could you please point me to a v.2.6 win32 build that contains an avidemux_Qt4.exe (and works on Windows XP) ;I have access to archive builds on this site http://sourceforge.net/projects/avidemux-mswin/files/beta/2.6/.

From searching the forum, I understood that a Qt4 executable can be launched from the command line. Is that right ? But will redirection of StandardOut and ErrorOut work in that case, so that I can capture a complete log of all what happens ? (I work in a Windows environment... ).

For the tests: give me a few hours and I'l be back with the result.

Edit 06h06 : forget my question about redirection of StdOut and ErrOut. I can always look in user\Application Data\avidemux\admlog.txt.

[SeventyPlus]

The buffer underflow warning are not really a problem
I'm more concerned about these ones

Code Select Expand


[AudioStream] Warning skew in dts =-55608000,
[AudioStream] Warning skew lastDts= 00:00:55,648 
[AudioStream] Warning skew newDts= 00:00:00,040   


Could you provide a couple of files which, when joined produce these messages ?
They dont need to be be long

Is there a "threshold" skew value from which on there is reason to be concerned ?
For example: are following skew values relevant for the case or do I have to search better examples (I am looking for the smallest files possible)

Code Select Expand


[switchToSegment] Switched ok to segment 0 (dontdecode=0)
[ADM_videoStreamCopy]  Fixating start time by 0
[ADM_videoStreamCopy]  Starting DTS=0, PTS=262 ms
[goToTime]  go to time 0.00 secs
[goToTime] => seg 0, rel time 0.00 secs
[AudioStream] Warning skew in dts =262222,
[AudioStream] Warning skew lastDts= 00:00:00,000 
[AudioStream] Warning skew newDts= 00:00:00,262   
[setupAudio] Setting up 1 audio track(s)
[setupAudio] [audioTrack 0] Creating audio encoding stream, starttime  00:00:00,000 (copy)
[goToTime]  go to time 0.00 secs
[goToTime] => seg 0, rel time 0.00 secs
[AudioStream] Warning skew in dts =262222,
[AudioStream] Warning skew lastDts= 00:00:00,000 
[AudioStream] Warning skew newDts= 00:00:00,262   
[FF] Muxer opened


EDIT 7h10 : I am not so certain you should be concerned about the 'Warning skew' messages.

The 'Warning skew' sample message I provided above comes when joining this files in this order:
(1) First source (load): file 2000-01-01_32.MPG
(2) Second source (append) : file 2000-01-01_33.MPG

But when I do the joining of the same files in the following order:
(1) First source (load): file 2000-01-01_33.MPG
(2) Second source (append) : file 2000-01-01_32.MPG
then the 'Warning skew' messages become as follows:

Code Select Expand


[AudioStream] Warning skew in dts =-4752000,
[AudioStream] Warning skew lastDts= 00:00:04,792 
[AudioStream] Warning skew newDts= 00:00:00,040


If you check the duration of file 2000-01-01_33.MPG (=4sec800)(*), then you will see that the 'skew' = duration of the first joined file if you inverse the "logical" (? whatever that means) order of the two files in a join.

The initial sample result  (the one with the 'skew' of 55 sec) comes from a set of files, provided to me by the owner of the files. The files in this test set were originally named 2000-01-01.MPG, 2000-01-01_01.MPG, 2000-01-01_02.MPG, etc.
To make the test set coherent, I renamed the 2000-01-01.MPG file to 2000-01-01_00.MPG (and I am not even certain these files were transfered from the same SD memory card). In this case, the 55 sec 'skew warning' corresponds to the duration of file 2000-01-01_00.MPG in the join-experiment; file 2000-01-01_00.MPG being the first file in the join.

(*) I still feel unhappy the way avidemux 2.6 expresses the 'duration' of file, example : 18.9422222 sec for a file of 466 frames at 25 fps (I am used to the simple arithmetics: for MPEG2 video, duration is calc NumberOfFrames/FPS)

If nevertheless you want to continue the 'skew warning' investigations, please let me know.

[Jan Gruuthuse]

2000-01-01_00.MPG
2000-01-01_01.MPG
2000-01-01_02.MPG

This naming scheme may cause the auto append function even when appending one file?

But what strikes me: at every append of a source file, the program seems to do a rescan of ALL similar .mpg source files as if the program is willing to do an automatic append.
So : if for example a directory contains 100 .mpg source files and I want to join (Load+append) these 100 .mpeg source files, than the program will do 100*100 rescans (=10 000). Does not seem performant (and will flood a log file or in my case a streamreader who will be "listening" for progress information).
Reminder: this rescan does not happen with the v2.5.6 version. In the log file of a similar run with v2.5.6 avidemux_cli.exe, you will find a message like "There was several files, but dont append was forced" and no more scans are done when each source file is appended.

Don't know if the developers already addressed/planned --append no  switch, stopping auto append. Issue raised here: batch processing: append

[mean]

A skew more than a few ms is a problem
The only case where it is "normal" is when the file are autoloaded because they are sequential (i.e. a001.mpg, a002.mpg) because in that case
no  timing correction is done as the file are considered to be one logicial file split into several physical files

[SeventyPlus]

A skew more than a few ms is a problem
The only case where it is "normal" is when the file are autoloaded because they are sequential (i.e. a001.mpg, a002.mpg) because in that case
no  timing correction is done as the file are considered to be one logicial file split into several physical files

May I conclude that when a user manually joins several files (whether yes or no they come from a set) in the order the user prefers, then this becomes a problem for avidemux ? Or will avidemux be capable to produce allways a correct mpeg program stream, whatever the input "order" of the source files ?

The response is very important for me, because the GUI application I am developping (to be used by someone else) is explicitly based on the principle that the user can freely pick which files he will join (even from different "sets" and directories) and freely choose the order in which the source files will be joined (provided all source files have identical characteristics).

[mean]

If you choose the files one per one, the timing will be automatically corrected
If you load 1 file and let it autoload the others, no timing correction is done

[SeventyPlus]

Problem status on 02 june 2012:

• With V2.6 avidemux_r7996_win32 (nightly) version, the CLI executable (avidemux_cli.exe) with tinyPy scripts now produces 'joined' program stream MPEG output files (.mpg) with correct video (no longer are 2 B-frames lost at the end of each segment, as what happened with V2.5.6).
  
• 
  I then concentrated on audio checks with the V2.6 r7996 win32 version, joining MPEG sources with 'plain stero' audio (= MPEG Audio Version 1 Layer 2, CBR, 256 kbps, sample rate 48 kHz, 2 channels) into an MPEG Program stream output.
  

This are my findings :

• With both CLI and GUI executables, when joining multiple mpeg source files (= pure copy, without transcoding neither video nor audio), audio is present in the beginning of the output but from a 'variable-and-I-cannot-find-the-pattern' moment on there is no more audio in the output.
• 
  When audio ceases to be present, it is always on an imput segment boundary, but never on the same boundary and never for the same input file (*).
  
• 
  No audio present means: there is no audio. To check that, I demultiplexed (with TMPGenc) the MPEG program stream output and imported the elementary audio stream in Audacity to check and measure the length.
  
• 
  The problem occurs at every 'join' test I did, both with CLI executable and GUI executable (also tested with the nigthly updated version of avidemux_r7996_win32 of 02-Jun-2012 06:46).
  
• 
  As I said: I cannot find a pattern, but I made certain to have a final test set of input source files with each individual source file having produced audio in the output during a previous test (where of course the sequence-order of the input files in the test-set was different).
  
• 
  I tested with the files in "ascending" and "descending" order (files are named 2000-01-01_nn.MPG, where nn is the variable) but no fixed pattern appears. At a moment I thought I found a pattern when all tests with test-sets of files in "descending" order ceased to produce audio after 7 segments (no audio more from the 8th segment on), but this pattern is broken with tests in "ascending" order (in tests with "ascending" ordered input files, there was no more audio in the output from the 4th segment on), but this may all be coincidence.
  
• 
  I also checked (with Audacity) if audio in the output was limited on a "total-time" boundary, but that does not seem the case (example of some "total audio durations" in the output: 43 sec, 1m05sec).
  

The log file of one test is huge (> 4Mbyte). Error lines that concern me, are like this:

Code Select Expand

[adm_lavLogCallback] [lavc] buffer underflow i=0 bufi=23855 size=25080
[adm_lavLogCallback] [lavc] buffer underflow i=0 bufi=794 size=9264
[adm_lavLogCallback] [lavc] buffer underflow i=0 bufi=2813 size=9264
[adm_lavLogCallback] [lavc] buffer underflow i=0 bufi=4832 size=9264
[adm_lavLogCallback] [lavc] buffer underflow i=0 bufi=6851 size=9264
[adm_lavLogCallback] [lavc] buffer underflow i=0 bufi=8870 size=9264
[adm_lavLogCallback] [lavc] buffer underflow i=1 bufi=42 size=768
[AudioStream] Warning skew in dts =-14832000,
[AudioStream] Warning skew lastDts= 00:00:14,872 
[AudioStream] Warning skew newDts= 00:00:00,040   
[adm_lavLogCallback] [lavc] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 3948319 >= 2615599
[saveLoop] [FF]Error writing audio packet
[adm_lavLogCallback] [lavc] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 3948319 >= 2617759
[saveLoop] [FF]Error writing audio packet
[adm_lavLogCallback] [lavc] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 3948319 >= 2619919
[saveLoop] [FF]Error writing audio packet

My request: is someone willing to look into this issue.

I can of course provide: the CLI command line + py script (or the saved job for a GUI 'join' test) and the corresponding log file.
But can someone tell me the procdure for providing a 200 kilobyte (rar compressed format) log file:

• Can such a 200 kbyte file be "attached" to a forum post ?
  
• Or is there an avidemux ftp server for this, where someone can put input to, and in that case: is there a 'ticket' procedure and rule for name-references of the input provided ?

(*) My audio-tests were 'contaminated' by the fact that V2.6 avidemux (both GUI and CLI) wrongly indexed one particular source file (the audio section is missing in the idx2 file). This one particular mpeg source file has audio in it and is correctly indexed by avidemux v2.5.6. For this v2.6 indexing problem, I prefer to start a different Post.

[mean]

Please use rapidshare or similar to upload large files.
Are you sure the files are loaded one per one and not auto-appent ?
The loss of audio is probably linked to the skew, i.e. discontinuous audio

[Jan Gruuthuse]

As mean says. You can check this if each video in that folder processed has it's own index (.idx2). While testing better to delete these, when these where made by a previous avidemux 2.6 (lower revison number as the one your testing)

[SeventyPlus]

Please use rapidshare or similar to upload large files.
Are you sure the files are loaded one per one and not auto-appent ?
The loss of audio is probably linked to the skew, i.e. discontinuous audio

Following information documents the problem that, when joining with avidemux v.2.6.0 r7996 (and previous builds) several MPEG2 Program stream source files (coming from a Panasonic SD video camera) into one single MPEG program stream output file, at some point there is no more audio in the output file (see Reply #24 for a description of the problem).

This documented test was done with the GUI program of avidemux v.2.6.0 r7996 win32 : all input source files where manually loaded one per one ('load' for the first source file, then 'append' for each other source file), no auto-append was done.
The join of the source files is a "raw" join (copy), no trimming of the input sources was done.

The characteristics of the input source files are as follows (information gathered with MediaInfo); all input source files have identical characteristics.

Example characteristics of input source file 2000-01-01_01.MPG (by MediaInfo)

Code Select Expand


General
Complete name                            : E:\PANASONIC_CAPTURES\STEP_3_JOIN_AVIDEMUX_AVI\VARIA1\2000-01-01_01.MPG
Format                                   : MPEG-PS
File size                                : 14.5 MiB
Duration                                 : 18s 648ms
Overall bit rate mode                    : Variable
Overall bit rate                         : 6 524 Kbps

Video
ID                                       : 224 (0xE0)
Format                                   : MPEG Video
Format version                           : Version 2
Format profile                           : Main@Main
Format settings, BVOP                    : Yes
Format settings, Matrix                  : Custom
Format settings, GOP                     : M=3, N=12
Duration                                 : 18s 640ms
Bit rate mode                            : Variable
Bit rate                                 : 6 138 Kbps
Maximum bit rate                         : 9 558 Kbps
Width                                    : 704 pixels
Height                                   : 576 pixels
Display aspect ratio                     : 1.738
Frame rate                               : 25.000 fps
Standard                                 : PAL
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Interlaced
Scan order                               : Top Field First
Compression mode                         : Lossy
Bits/(Pixel*Frame)                       : 0.605
Stream size                              : 13.6 MiB (94%)
Color primaries                          : BT.470-2 System B, BT.470-2 System G
Transfer characteristics                 : BT.470-2 System B, BT.470-2 System G
Matrix coefficients                      : BT.470-2 System B, BT.470-2 System G

Audio
ID                                       : 192 (0xC0)
Format                                   : MPEG Audio
Format version                           : Version 1
Format profile                           : Layer 2
Duration                                 : 18s 648ms
Bit rate mode                            : Constant
Bit rate                                 : 256 Kbps
Channel(s)                               : 2 channels
Sampling rate                            : 48.0 KHz
Compression mode                         : Lossy
Stream size                              : 583 KiB (4%)


The content of the project file (automated generated tinyPi script), created by the GUI program avidemux V2.6 rr7996 win32 for this particular 'copy' join test is as follows:

Code Select Expand


#PY  <- Needed to identify#
#--automatically built--
#--Project: C:/My Programs and Doc/MyMpegJoiner/TEMPLATES/JoinScript_Files01+02+75-81+00_v26r7776_Project.py

adm=Avidemux()
#** Video **
# 10 videos source
adm.loadVideo("E:/PANASONIC_CAPTURES/STEP_3_JOIN_AVIDEMUX_AVI/VARIA1/2000-01-01_01.MPG")
adm.appendVideo("E:/PANASONIC_CAPTURES/STEP_3_JOIN_AVIDEMUX_AVI/VARIA1/2000-01-01_02.MPG")
adm.appendVideo("E:/PANASONIC_CAPTURES/STEP_3_JOIN_AVIDEMUX_AVI/VARIA1/2000-01-01_75.MPG")
adm.appendVideo("E:/PANASONIC_CAPTURES/STEP_3_JOIN_AVIDEMUX_AVI/VARIA1/2000-01-01_76.MPG")
adm.appendVideo("E:/PANASONIC_CAPTURES/STEP_3_JOIN_AVIDEMUX_AVI/VARIA1/2000-01-01_77.MPG")
adm.appendVideo("E:/PANASONIC_CAPTURES/STEP_3_JOIN_AVIDEMUX_AVI/VARIA1/2000-01-01_78.MPG")
adm.appendVideo("E:/PANASONIC_CAPTURES/STEP_3_JOIN_AVIDEMUX_AVI/VARIA1/2000-01-01_79.MPG")
adm.appendVideo("E:/PANASONIC_CAPTURES/STEP_3_JOIN_AVIDEMUX_AVI/VARIA1/2000-01-01_80.MPG")
adm.appendVideo("E:/PANASONIC_CAPTURES/STEP_3_JOIN_AVIDEMUX_AVI/VARIA1/2000-01-01_81.MPG")
adm.appendVideo("E:/PANASONIC_CAPTURES/STEP_3_JOIN_AVIDEMUX_AVI/VARIA1/2000-01-01_00.MPG")
#10 segments
adm.clearSegments()
adm.addSegment(0,0,18942222)
adm.addSegment(1,0,10080000)
adm.addSegment(2,0,14880000)
adm.addSegment(3,0,13440000)
adm.addSegment(4,0,10560000)
adm.addSegment(5,0,5982222)
adm.addSegment(6,0,7902222)
adm.addSegment(7,0,5982222)
adm.addSegment(8,0,5502222)
adm.addSegment(9,0,55680000)
adm.markerA=0
adm.markerB=148951110

#** Postproc **
adm.setPostProc(0,0,0)

#** Video Codec conf **
adm.videoCodec("Copy")

#** Filters **

#** Audio **
adm.audioClearTracks()

#** Track 0 **
adm.audioAddTrack(0)
adm.audioCodec(0,"copy");
adm.audioSetPal2Film(0,0)
adm.audioSetFilm2Pal(0,0)
adm.audioSetNormalize(0,0,0)

#** Muxer **
adm.setContainer("ffPS","muxingType=2","acceptNonCompliant=False","muxRatekBits=11000","videoRatekBits=9800","bufferSizekBytes=224")

#End of script


This project file shows in which order the source files were loaded. The join of the source files is a "raw" join (copy), no trimming of the input sources was done.
In the output MPEG program stream .MPG file, audio is present for the 3 first segments (segment #0 2000-01-01_01.MPG, segment #1 2000-01-01_02.MPG and segment #2 2000-01-01_75.MPG), but audio is missing in the output starting from segment #3 (2000-01-01_75.MPG) and this until the end of the output file.

Each individual source file has audio and, when joined in a different order, the audio of that source file may (or may not) be present in the output.

The log file produced by avidemux GUI V2.6 r7996 win32 GUI for this particular test (admlog.txt) is available for download at this URL (available for 30 days): http://home.scarlet.be/hvda/avidemux01/ (log file in rar packed size: 200KB, unpacked: 4MB)
[EDIT August 14 2012 : download no longer available]

Similar tests, done with avidemux_cli v2.6r7996 win32 CLI (instead of GUI), produce similar results : at some point in the output file, there is no more audio.
The difference between the GUI generated script (project file) and my CLI script files is : in my CLI script file there is no 'clearSegments', no 'addSements', no 'markerA' and no 'markerB' commands.

Remarks:

• The same source files can be joined with avidemux V2.5.6 without audio loss in the output.
  
• 
  These files will certainly have different timings, but the purpose of these tests is:  to evaluate the capability of avidemux to join source files where "the user can freely pick which files he will join (even from different "sets" and directories) and freely choose the order in which the source files will be joined (provided all source files have identical characteristics)". So if there are timing differences between source file, my understanding is that "If you choose the files one per one, the timing will be automatically corrected" (by avidemux).

I hope that this information may contribute in finding a correction for this issue.

EDIT: 03 june 2012 12h10 : same result as described above when test done with 'nightly' r79997 win32

[Jan Gruuthuse]

Looks like from where you selected the file to append, it auto appends the remaining video parts that follow in sequence and by doing so not making the a seperate .idx2 for the appended video?

Can't change the --append behavior as it has no parameter to auto append or not?
Unless you would try to not append in the job (.py)
But append each file at a time from script that passes filename to command line that runs job.py?
Also try to build the index file at this stage with --index-mpeg.
Don't know if this will work as expected?
Reduce your folder only to 10 video clips for testing. makes test shorter and probably no memory problems in this stage.

command line usage

[SeventyPlus]

Looks like from where you selected the file to append, it auto appends the remaining video parts that follow in sequence and by doing so not making the a seperate .idx2 for the appended video?

Can't change the --append behavior as it has no parameter to auto append or not?
Unless you would try to not append in the job (.py)
But append each file at a time from script that passes filename to command line that runs job.py?
Also try to build the index file at this stage with --index-mpeg.
Don't know if this will work as expected?
Reduce your folder only to 10 video clips for testing. makes test shorter and probably no memory problems in this stage.

command line usage

Jan,
I would like to remind that the documented test was not done by starting an avidemux executable (neither avidemux3.exe nor avidemux_cli.exe) from the command line but was done with the interactive avidemux GUI version, and that :

all input source files where manually loaded one per one ('load' for the first source file, then 'append' for each other source file), no auto-append was done.

I preferred to submit the test-result of a plain GUI test, rather than a CLI test, in order to cut short on all eventual discussions on whether my own py scripts are good or not, so that we can keep concentrated on the real problem.
We should also keep in mind that:

The same source files can be joined with avidemux V2.5.6 without audio loss in the output.