Avidemux Forum

Avidemux => Windows => Topic started by: avilon on January 30, 2021, 08:04:32 PM

Title: How to append 500 mp4 files into single video?
Post by: avilon on January 30, 2021, 08:04:32 PM
Avidemux 2.7.5

I have 1645 dash camera mp4 clips 60 sec. each, named sequentially GRMN0010 - GRMN1655.
How to append these in 300-400 files in batches (by trip days).?
Drag and drop likely isn't reliable as there is no easy way to check result.

Thanks for help.
Title: Re: How to append 500 mp4 files into single video?
Post by: butterw on January 30, 2021, 10:16:07 PM
1. you should put your file batches in separate folders.
2. you should then run a .py script (project) which loads all .mp4 file in a folder

what processing if any do you intend to do ?
 
Title: Re: How to append 500 mp4 files into single video?
Post by: avilon on January 30, 2021, 10:34:39 PM
I must rotate 180 degrees and perhaps experiment decreasing bit rate total 90GB of footage.
But I'd leave this as a second step.

I have some unsuccessful experiences with .py script.
I thought there might some easier solution, like loading from text file list in correct order.
Title: Re: How to append 500 mp4 files into single video?
Post by: butterw on January 30, 2021, 11:10:43 PM
It's easy as pie ;), I'll post the loading script in the patch submission subforum.
You can do the rest of the processing manually.

I would recommend you save your project settings when you've processed your first folder, so you can apply it to the other folders.
 
Title: Re: How to append 500 mp4 files into single video?
Post by: butterw on January 31, 2021, 12:00:39 AM
Here's a loading script:
https://github.com/butterw/bShaders/raw/master/test_LimitedRange/dirLoad.py
Title: Re: How to append 500 mp4 files into single video?
Post by: eumagga0x2a on January 31, 2021, 12:21:25 AM
If it is preferable to append without pre-sorting files into different directories, then the following should do it with some graphical configuration:

# set a few important variables
ext = "mp4"
sep = "\\"
prefix = "GRMN"
# instantiate
adm = Avidemux()
gui = Gui()
# choose input directory
src = gui.dirSelect("Select the source folder")
if src is None:
    return 0
# create a list of all files with extension ext there
list = get_folder_content(src,ext)
if list is None:
    gui.displayError("Error","No " + ext + " files found in \"" + src + "\"")
    return 0
# create configuration dialog
spinStart = DFInteger("Start at file no.",0,9999)
spinStart.value = 10
spinNumber = DFInteger("Number of files to process:",1,500)
spinNumber.value = 300
dialog = DialogFactory("Multi-Append Setup")
dialog.addControl(spinStart)
dialog.addControl(spinNumber)
if 1 != dialog.show():
    return 0
# is there a sane method to format a string in tinyPy?
start = spinStart.value
number = spinNumber.value
appended = 0
for i in range(start,start+number):
    filename = prefix
    if i < 10:
        filename += "000"
    elif i < 100:
        filename += "00"
    elif i < 1000:
        filename += "0"
    filename += str(i) + "." + ext
    for e in list:
        if filename == basename(e):
            if not appended:
                if not adm.loadVideo(src + sep + filename):
                    gui.displayError("Error","Cannot load \"" + filename + "\"")
                    return 0
            elif not adm.appendVideo(src + sep + filename):
                gui.displayError("Error","Cannot append \"" + filename + "\"")
                return 0
            appended += 1
# done
if not appended:
    gui.displayError("Error","No files loaded")
    return 0
gui.displayInfo("The End",str(appended) + " files appended")
return 1

Important: please ensure hw accelerated decoding (on Windows: DXVA2) is disabled prior to appending a large number of files.
Title: Re: How to append 500 mp4 files into single video?
Post by: butterw on January 31, 2021, 10:13:51 AM
There will be limit to what you can load and edit confortably in a single batch (>5 hours footage per day, huge filesize), so it may be necessary to break it down further and select the starting file and the number of files which will be processed in a folder. Start low for testing, save your project and don't delete the source files.
The issue with splitting the processing is that you have to keep track of the starting file for each run, and have to deal with split time.
Title: Re: How to append 500 mp4 files into single video?
Post by: butterw on January 31, 2021, 05:02:16 PM
if not adm.appendVideo(fname): raise("! Cannot append ")

!!! On Win10, testing append with mp4 h264 with non-compatible dimensions, the TinyPy raise causes a crash here:
After the tinyPy exception is displayed, I get a Unknown runtime error.. Occurs with both DXVA2 and OpenGL

 
Title: Re: How to append 500 mp4 files into single video?
Post by: eumagga0x2a on January 31, 2021, 05:16:10 PM
What does admlog.txt say?
Title: Re: How to append 500 mp4 files into single video?
Post by: butterw on January 31, 2021, 05:20:47 PM
Quote from: eumagga0x2a on January 31, 2021, 05:16:10 PMWhat does admlog.txt say?

[Script] Tinypy INFO -

[Script] Tinypy INFO - import_fname
[Script] Tinypy INFO -
 
[Script] Tinypy INFO -     exec(code,g)
[Script] Tinypy INFO -

[Script] Tinypy INFO - ?
[Script] Tinypy INFO -
 
[Script] Tinypy INFO - if not adm.appendVideo("B:/mpc/a.mp4"): raise("Cannot append")
[Script] Tinypy INFO -

[Script] Tinypy INFO -
Exception:

[Script] Tinypy INFO - Cannot append
[Script] Tinypy INFO -

[draw] 19:19:28-929 D3D : Draw!
[refresh] 19:19:28-929 Refresh**
[draw] 19:19:28-952 D3D : Draw!
[refresh] 19:19:28-952 Refresh**
Saving crash file to B:\mpc\avidemux_64\settings\crash.py
Scripting video streams
Scripting segments
Scripting markers
Scripting post-processing
Scripting video encoder
Scripting video filters
Scripting audio tracks
Scripting muxer
RAX: 67204939  RBX: C0000028  RCX: 00000000  RDX: 028C9CB0  RSI: 008A9278  RDI: 6A8A3928  RSP: 008A8B40  RBP: 008A8C40
R8: 00000001  R9: 00000000  R10: 00000000  R11: 0040041E  R12: 6A820000  R13: 008A9820  R14: 6A891A2A  R15: 008A97C0
RIP: 6C6E2B10  EFlags: 00000202

UNKNOWN
RtlRaiseStatus  [ntdll.dll]
Crash Dump for ExceptionFilter
UNKNOWN
RtlRaiseStatus  [ntdll.dll]
ADM_setCrashHook  [libADM_core6.dll]
ADM_setCrashHook  [libADM_core6.dll]
ADM_setCrashHook  [libADM_core6.dll]
UnhandledExceptionFilter  [KERNELBASE.dll]
memset  [ntdll.dll]
_C_specific_handler  [ntdll.dll]
_chkstk  [ntdll.dll]
RtlWalkFrameChain  [ntdll.dll]
RtlRaiseStatus  [ntdll.dll]
memset  [ntdll.dll]
setjmpex  [msvcrt.dll]
unknown function  [libADM_script_tinyPy.dll]
unknown function  [libADM_script_tinyPy.dll]
unknown function  [libADM_script_tinyPy.dll]
unknown function  [libADM_script_tinyPy.dll]
unknown function  [libADM_script_tinyPy.dll]
unknown function  [libADM_script_tinyPy.dll]
unknown function  [libADM_script_tinyPy.dll]
unknown function  [libADM_script_tinyPy.dll]
unknown function  [libADM_script_tinyPy.dll]
unknown function  [libADM_script_tinyPy.dll]
unknown function  [libADM_script_tinyPy.dll]
unknown function  [libADM_script_tinyPy.dll]
unknown function  [libADM_script_tinyPy.dll]
createEngine  [libADM_script_tinyPy.dll]
A_appendVideo(char const*)  [avidemux_portable.exe]
MainWindow::scriptFileActionHandler()  [avidemux_portable.exe]
MainWindow::qt_static_metacall(QObject*, QMetaObject::Call, int, void**)  [avidemux_portable.exe]
QMimeType::qt_static_metacall(QObject*, QMetaObject::Call, int, void**)  [Qt5Core.dll]
QAction::activate(QAction::ActionEvent)  [Qt5Widgets.dll]
QMenu::setIcon(QIcon const&)  [Qt5Widgets.dll]
QMenu::leaveEvent(QEvent*)  [Qt5Widgets.dll]
QMenu::mouseReleaseEvent(QMouseEvent*)  [Qt5Widgets.dll]
QWidget::event(QEvent*)  [Qt5Widgets.dll]
QMenu::event(QEvent*)  [Qt5Widgets.dll]
QApplicationPrivate::notify_helper(QObject*, QEvent*)  [Qt5Widgets.dll]
QApplication::notify(QObject*, QEvent*)  [Qt5Widgets.dll]
QCoreApplication::sendSpontaneousEvent(QObject*, QEvent*)  [Qt5Core.dll]
QApplicationPrivate::sendMouseEvent(QWidget*, QMouseEvent*, QWidget*, QWidget*, QWidget**, QPointer<QWidget>&, bool, bool)  [Qt5Widgets.dll]
QDesktopWidget::qt_metacall(QMetaObject::Call, int, void**)  [Qt5Widgets.dll]
QDesktopWidget::qt_metacall(QMetaObject::Call, int, void**)  [Qt5Widgets.dll]
QApplicationPrivate::notify_helper(QObject*, QEvent*)  [Qt5Widgets.dll]
QApplication::notify(QObject*, QEvent*)  [Qt5Widgets.dll]
QCoreApplication::sendSpontaneousEvent(QObject*, QEvent*)  [Qt5Core.dll]
QGuiApplicationPrivate::processMouseEvent(QWindowSystemInterfacePrivate::MouseEvent*)  [Qt5Gui.dll]
QGuiApplicationPrivate::processWindowSystemEvent(QWindowSystemInterfacePrivate::WindowSystemEvent*)  [Qt5Gui.dll]
QWindowSystemInterface::sendWindowSystemEvents(QFlags<QEventLoop::ProcessEventsFlag>)  [Qt5Gui.dll]
QEventDispatcherWin32::processEvents(QFlags<QEventLoop::ProcessEventsFlag>)  [Qt5Core.dll]
qt_plugin_instance  [qwindows.dll]
QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>)  [Qt5Core.dll]
QCoreApplication::exec()  [Qt5Core.dll]
UI_RunApp()  [avidemux_portable.exe]
startAvidemux(int, char**)  [avidemux_portable.exe]
SDL_main  [avidemux_portable.exe]
ms2timedisplay(unsigned int)  [avidemux_portable.exe]
unknown function  [avidemux_portable.exe]
unknown function  [avidemux_portable.exe]
BaseThreadInitThunk  [KERNEL32.DLL]
RtlUserThreadStart  [ntdll.dll]

EDIT: Doesn't crash in v2.7.1 x64 VC++, which doesn't display the error message about incompatible dimensions on append.

EDIT2: adm.appendVideo(fname) doesn't crash, the issue seems to be with raise.
Title: Re: How to append 500 mp4 files into single video?
Post by: eumagga0x2a on January 31, 2021, 07:07:43 PM
Please let me have a look at the part of the log file reflecting the steps leading to the crash, not just the record of the aftermath.

createEngine, the last call before the first error, is normally used only during the init of the GUI, so that I cannot tell from this info what is going on here and also cannot reproduce such a crash on Linux. You got the error message about size mismatch, didn't you?

edit: just saw your EDIT2. Looks plausible.
Title: Re: How to append 500 mp4 files into single video?
Post by: eumagga0x2a on January 31, 2021, 07:41:02 PM
Cannot reproduce on Windows 10 using any of the latest nightlies.
Title: Re: How to append 500 mp4 files into single video?
Post by: butterw on January 31, 2021, 07:50:06 PM
I've added the log chunk before, but I doubt there is anything useful.
I get the 2 normal error messages: size mismatch error, followed by append error.
If raise is used here, I then get the tinyPy Exception followed by the crash messages.

adm = Avidemux()
if not adm.loadVideo("B:/mpc/1.mp4"): raise("Cannot load")
if not adm.appendVideo("B:/mpc/a.mp4"): raise("Cannot append")
# 1 is 1280x720, a is 1302x704

EDIT: reproducible with other mismatched files.
Title: Re: How to append 500 mp4 files into single video?
Post by: eumagga0x2a on January 31, 2021, 07:54:29 PM
Thanks, does

raise("panic"")
alone trigger the crash on your system (it doesn't on mine)? Are you sure you didn't mix different builds or such?
Title: Re: How to append 500 mp4 files into single video?
Post by: butterw on January 31, 2021, 08:04:33 PM
Quote from: eumagga0x2a on January 31, 2021, 07:54:29 PMalone trigger the crash on your system (it doesn't on mine)? Are you sure you didn't mix different builds or such?

EDIT: raise("panic") doesn't crash on it's own, but it does quite easily crash. Ex after the append size mismatch error message, but not only:

adm = Avidemux()
adm.loadVideo("B:/mpc/1.mp4")
raise("panic")

EDIT2: the issue isn't present with 2.7.1 VC++ where there wasn't the TinyPy exception popup, but is with both latest Vc++ and mingw x64 win 2.7.7 dev builds.
Title: Re: How to append 500 mp4 files into single video?
Post by: eumagga0x2a on January 31, 2021, 08:26:18 PM
It doesn't make much sense to compare to ancient 2.7.1, couldn't make any of the both latest nightlies crash on my Windows 10 installation upon multiple attempts with different raise() invocations.
Title: Re: How to append 500 mp4 files into single video?
Post by: butterw on January 31, 2021, 08:37:00 PM
It didn't crash with disabled config3.

Does Tinypy raise get called outside of user scripts ?
Title: Re: How to append 500 mp4 files into single video?
Post by: butterw on January 31, 2021, 08:50:32 PM
Quote from: butterw on January 31, 2021, 08:37:00 PMIt didn't crash with disabled config3.

Does Tinypy raise get called outside of user scripts ?

Try with "reset_encoder_on_video_load" : true
I think this causes the issue.
Title: Re: How to append 500 mp4 files into single video?
Post by: eumagga0x2a on January 31, 2021, 08:57:11 PM
Quote from: butterw on January 31, 2021, 08:50:32 PMTry with "reset_encoder_on_video_load" : true

A good idea, I haven't tried this yet, will give it a go as soon as my Windows installation is available again.
Title: Re: How to append 500 mp4 files into single video?
Post by: eumagga0x2a on January 31, 2021, 09:16:28 PM
No luck ( = no crash) so far.
Title: Re: How to append 500 mp4 files into single video?
Post by: dosdan on February 01, 2021, 12:35:19 AM
Simple, repetitive tasks like this may be better suited to FFMPEG.

I've already explained about working at the command-prompt window in https://avidemux.org/smif/index.php/topic,19416.msg90907.html#msg90907

To that, I'll add that you can use the Tab key to autocomplete directory names and filenames e.g. type c and press Tab to find the first match of a name, in the current directory, beginning with c. Type co to find a name beginning with co. If it's not the match you want, press Tab again to bring up the next match. You can also use the Up Arrow to bring back previous command lines for direct reuse or alteration/reuse.  Tab and Up Arrow make the command line a much friendlier place.

For those who would like to try FFMPEG concatenation, but don't have suitable media files, here is a 60s 720p60 MP4 (43MB) which you can d/l to test this. You can test audio syncing using the referee whistle blown at the start or with sound of the "thunks" when the ball is kicked.

60s_testfile.mp4 (https://dl.dropbox.com/s/anlyoqxk5w2edea/60s_testfile.mp4)

Here is a batchfile to create numbered copies of this file. It's currently set to makes 10 copies, GRMN0010.mp4 to GRMN0019.mp4. You can easily alter this to make up to 10,000 numbered copies (0,1,9999).  Note: use Ctrl-C if you wish to abort the operation of a batchfile.

Create_numbered_copies.bat
@echo off
rem SETLOCAL EnableDelayedExpansion
for /L %%i IN (10,1,19) do call :docopy %%i
goto :EOF

:docopy
set FN=000%1
set FN=%FN:~-4%
copy 60s_testfile.mp4 GRMN%FN%.mp4

Here is the batchfile to concatenate the MP4s. I suggest you start with 10 x 60s GRMN MP4 files in the current directory.

concat_grmn_mp4s.bat
@echo off
rem setlocal EnableDelayedExpansion
cls

if exist journey_list.txt del journey_list.txt

(for %%G in (GRMN*.mp4) do @echo file '%%G') >> journey_list.txt
rem if exist journey_list.txt start /w journey_list.txt

ffmpeg -f concat -safe 0 -i journey_list.txt -c copy -hide_banner -y output.mp4
rem ffmpeg -f concat -safe 0 -i journey_list.txt -vf "hflip,vflip,format=yuv420p" -metadata:s:v rotate=0  -c:a copy -c:v h264 -b:v 5M -y -hide_banner output.mp4

FFMPEG expects the filenames in the list to be specified in a certain format. Here is the contents of journey_list.txt
file 'GRMN0010.mp4'
file 'GRMN0011.mp4'
file 'GRMN0012.mp4'
file 'GRMN0013.mp4'
file 'GRMN0014.mp4'
file 'GRMN0015.mp4'
file 'GRMN0016.mp4'
file 'GRMN0017.mp4'
file 'GRMN0018.mp4'
file 'GRMN0019.mp4'

Note: I believe the default sort-order under Windows, on both NTFS and FAT32-formatted drives, is alphabetic name-order, so the list should be sorted correctly because the numbers in the filenames have leading zeros. Without leading zeros, GRMN10.mp4 would be listed before GRMN2.mp4.

Since I don't know the options that Garmin uses to produce the AVC video stream in its MP4 files, you should test the audio sync and the cleanliness of the inter-file joining, toward the end of the concatentated output.mp4, say at the 9 min mark, to ensure that this MP4 concat method works OK for you.

Once you are happy with the 10 MP4 concat, you should work out the video bitrate to use for the recompressed 180°-rotated video stream. The -vf indicates that what follows is one or more video filters. When you use video filters in FFMPEG you can't just copy the video stream - you have to recompress it. So during the recompression process, you can compress the video stream a bit harder to reduce its size. For reasons of speeding up testing I suggest you work with just a single MP4 in the current directory during this determination process. You then deactivate the 1st FFMPEG line by remarking it out, i.e. by placing rem at the start of the line, activate the 2nd FFMPEG line by removing its rem, and save the changes. (Ctrl-S in many text editor programs).

The video stream bitrate is currently set to approx. 5Mbits/s. You can use M/m (Megabits/million bits), K/k (e.g. 5000K) and decimals (5.5M)

You mentioned grouping the files by day. As mentioned, you could split up the 1645 MP4s into different directories, one for each day. Or, if you know which sequences belong to which day you can work with them all in the current directory. To do that:

1. unREM the if exist journey_list.txt start /w journey_list.txt line. start /w starts a new program in a new window, here the associated default program for text files, (usually Notepad), and waits for you to finish editing journey_list.txt. (Without the /w option, the batchfile would continue to the next line as soon as this line was executed.)

2. You can now mark and delete a large block of filenames, leaving only the ones you want. Save the changes and close this window.

(https://dl.dropbox.com/s/xaaxhj8l8j6w3ll/working%20with%20concatenation.jpg)

3. Once this day's output.mp4 has been created, you should rename it to say day1.mp4 so it won't get overwritten when you later go to work on the next day's batch of MP4s.

Dan.
Title: Re: How to append 500 mp4 files into single video?
Post by: butterw on February 01, 2021, 10:10:35 AM
@dosdan
Avidemux is essentially a ffmpeg gui (with extras) + an editor/preview.

If the user wants to learn to handle the complex ffmpeg command-line (muxers+filters+encoders, batch, error-handling, build options) and the task doesn't require a GUI or preview, yes it's possible and in some cases more effective to use ffmpeg directly.

In this threads case, the single timeline preview of Avidemux can be used to review the footage. You don't get that option with ffmpeg.
Title: Re: How to append 500 mp4 files into single video?
Post by: eumagga0x2a on February 01, 2021, 10:31:50 AM
Quote from: butterw on January 31, 2021, 08:37:00 PMDoes Tinypy raise get called outside of user scripts ?

It is used in auto-generated scripts, please see avidemux_plugins/ADM_scriptEngines/tinyPy/src/PythonScriptWriter.cpp

Quote from: butterw on January 31, 2021, 08:37:00 PMIt didn't crash with disabled config3.

Have you preserved a copy of that config3 file?
Title: Re: How to append 500 mp4 files into single video?
Post by: eumagga0x2a on February 01, 2021, 10:34:06 AM
Quote from: butterw on February 01, 2021, 10:10:35 AMAvidemux is essentially a ffmpeg gui (with extras) + an editor/preview.

The main difference is that Avidemux uses its own demuxers, which is responsible for a lot of limitations but also advantages.
Title: Re: How to append 500 mp4 files into single video?
Post by: butterw on February 01, 2021, 11:00:13 AM
Quote from: butterw on January 31, 2021, 08:04:33 PMadm = Avidemux()
adm.loadVideo("B:/mpc/1.mp4")
raise("panic")
I'm confirming the issue on my end on the Jan18/2021 mingw win64 2.7.7 portable (on Win10).

After a clean start (power-on the PC)   
- doesn't crash with "reset_encoder_on_video_load" : false
- crashes with "reset_encoder_on_video_load" : true
Other settings at default.

I may try this on a clean 2.7.6 VC++ install

I've been getting quite a few of these slow crashes (TinyPy exception, saves crash.py and displays crash messages popups).
Title: Re: How to append 500 mp4 files into single video?
Post by: eumagga0x2a on February 01, 2021, 11:58:12 AM
Quote from: butterw on February 01, 2021, 11:00:13 AMI'm confirming the issue on my end on the Jan18/2021 mingw win64 2.7.7 portable (on Win10).

The latest one is from Jan 29.

Quote from: butterw on February 01, 2021, 11:00:13 AM- doesn't crash with "reset_encoder_on_video_load" : false
- crashes with "reset_encoder_on_video_load" : true

Please post your defaultSetting.py
Title: Re: How to append 500 mp4 files into single video?
Post by: butterw on February 01, 2021, 06:57:50 PM
Quote from: eumagga0x2a on February 01, 2021, 11:58:12 AM
Quote from: butterw on February 01, 2021, 11:00:13 AMI'm confirming the issue on my end on the Jan18/2021 mingw win64 2.7.7 portable (on Win10).

The latest one is from Jan 29.

Quote from: butterw on February 01, 2021, 11:00:13 AM- doesn't crash with "reset_encoder_on_video_load" : false
- crashes with "reset_encoder_on_video_load" : true

Please post your defaultSetting.py

tested with clean install of VsWin64 Jan 29 2021.
"reset_encoder_on_video_load" : true
It crashes if defaultSetting.py exists and isn't an empty file.

it crashes with this (I've also tried with the file generated by save settings as defaults):
#PY  <- Needed to identify #
#--automatically built--

adm = Avidemux()#crashes will this line commented also.
 
Title: Re: How to append 500 mp4 files into single video?
Post by: eumagga0x2a on February 01, 2021, 08:24:21 PM
Oh yes, no need for defaultSetting.py, just enter # into the shell and evaluate – tinyPy is completely broken afterwards. I could crash the latest VC++ build this way, failed to crash my own MinGW build, but the latter might have been just by accident. Looks like a bigger problem with script parsing within tinyPy.

Thank you.
Title: Re: How to append 500 mp4 files into single video?
Post by: eumagga0x2a on February 01, 2021, 08:38:24 PM
Quote from: eumagga0x2a on January 31, 2021, 12:21:25 AMIf it is preferable to append without pre-sorting files into different directories, then the following should do it with some graphical configuration:

# set a few important variables
ext = "mp4"
sep = "\\"
prefix = "GRMN"
# instantiate
adm = Avidemux()
gui = Gui()
# choose input directory
src = gui.dirSelect("Select the source folder")
if src is None:
    return 0
# create a list of all files with extension ext there
list = get_folder_content(src,ext)

(...... and so on .......)

Scratch that. I just realized that get_folder_content() is hardcoded to 200 entries max.
Title: Re: How to append 500 mp4 files into single video?
Post by: butterw on February 01, 2021, 10:22:36 PM
Quote from: eumagga0x2a on February 01, 2021, 08:38:24 PMScratch that. I just realized that get_folder_content() is hardcoded to 200 entries max.

Is it possible to raise the limit to 500 or 1000files or is this a TinyPy limitation ?
Title: Re: How to append 500 mp4 files into single video?
Post by: butterw on February 02, 2021, 10:26:49 AM
Using Win10 drag and drop, I'm crashing (exception filter error, crash.py) at about 230 jpg files. It also crashes, if I open less than 200 files, then attempt to append more.

So it seems currently Avidemux can't handle much more than 199 files.

 
Title: Re: How to append 500 mp4 files into single video?
Post by: eumagga0x2a on February 02, 2021, 11:36:43 AM
Did you test with hw accelerated video decode (DXVA2) enabled or disabled? Please disable and reduce editor cache size to minimum ( 8 pictures), restart Avidemux and retry.
Title: Re: How to append 500 mp4 files into single video?
Post by: eumagga0x2a on February 02, 2021, 12:32:20 PM
On Linux, with the undelying buildDirectoryContent modified to take a pointer to std::vector<std::string> as argument, I could successfully append 209 mp4 videos at 720p resolution.

But really, Avidemux consumes a ton of memory for all the editor caches and decoder instances, 5,5 GiB currently on my PC. At 1080p and with even more files, one would need probably a computer with 32 GiB of memory.
Title: Re: How to append 500 mp4 files into single video?
Post by: eumagga0x2a on February 02, 2021, 12:37:02 PM
Quote from: butterw on February 01, 2021, 10:22:36 PMIs it possible to raise the limit to 500 or 1000files or is this a TinyPy limitation ?

This is an arbitrary cap, the design of the function retrieving the content of a directory needed that. At the time as this code was written, this value was probably seen as more than sufficient.

The practical limitation is available memory as Avidemux creates a new decoder instance and a new editor cache for each reference video.
Title: Re: How to append 500 mp4 files into single video?
Post by: butterw on February 02, 2021, 12:55:37 PM
Quote from: butterw on February 02, 2021, 10:26:49 AMUsing Win10 drag and drop, I'm crashing (exception filter error, crash.py) at about 230 jpg files. It also crashes, if I open less than 200 files, then attempt to append more.

So it seems currently Avidemux can't handle much more than 199 files.
 
This was with the default dxva2 HW-Accel enabled. But the result was the same with it disabled.

I could go higher by lowering the editor cache preference from 16 to 8 decoded pictures.

Memory usage was not high (80MB), but it still crashed.
Title: Re: How to append 500 mp4 files into single video?
Post by: eumagga0x2a on February 02, 2021, 01:14:47 PM
Images are a special case if their filenames are sequential. Were they random in your test?

Else drag'n'dropping N sequentially named .jpg files will load N + (N-1) + (N-2) + ... = N(N+1)/2 files, equal 20100 when N=200, easily hitting the operating system limit on open files.
Title: Re: How to append 500 mp4 files into single video?
Post by: eumagga0x2a on February 02, 2021, 03:14:01 PM
I've pushed a pretty massive patch (https://github.com/mean00/avidemux2/commit/58eb29b45e265fcebc52aab768c09b3bc348931f) which removes the cap.
Title: Re: How to append 500 mp4 files into single video?
Post by: butterw on February 02, 2021, 04:12:57 PM
Quote from: eumagga0x2a on February 02, 2021, 01:14:47 PMImages are a special case if their filenames are sequential. Were they random in your test?

Else drag'n'dropping N sequentially named .jpg files will load N + (N-1) + (N-2) + ... = N(N+1)/2 files, equal 20100 when N=200, easily hitting the operating system limit on open files.

Image names were sequential and unknown to me Avidemux has a special built-in loader for image sequences which creates a single segment (! don't use drag-and-drop to open a long image sequence or only on the first image of the sequence). I tried remaming to .avi but got the same result.
 
Title: Re: How to append 500 mp4 files into single video?
Post by: eumagga0x2a on February 02, 2021, 04:38:08 PM
Quote from: butterw on February 02, 2021, 04:12:57 PMI tried remaming to .avi but got the same result.

The only one case where Avidemux relies on extension when loading files are VapourSynth scripts (vpy).
Title: Re: How to append 500 mp4 files into single video?
Post by: butterw on February 02, 2021, 05:45:33 PM
Using jpg extension for the first 199 files and jpeg for the rest did break-up the image sequence loading though.


I performed the test again this time with a single 10s mp4 file, which I appended over and over again until it crashed, which I can confirm occurs when it runs out of physical memory (VSWin64 v2.7.7 (210129_). For me, this occured before 300 files, but I'm running with 8MB of RAM. A 32GB machine should be able to open in excess of 1000 files this way (of course this also depends on the uncompressed size of the video).

If instead of trying to append all files sequentially, I save to file in Copy mode after loading 100 files, a much longer video sequence could be opened for the same amount of installed RAM. However, it might be simpler to just concatenate 100 files into 1 with ffmpeg, before loading in Avidemux.

The TinyPy crash with 200 files in a directory needs to be addressed, but I'm not sure it is practical to edit videos with more than 200 segments in Avidemux, there seems to be a lot of associated overhead.
Title: Re: How to append 500 mp4 files into single video?
Post by: eumagga0x2a on February 02, 2021, 07:38:31 PM
Quote from: butterw on February 02, 2021, 05:45:33 PMUsing jpg extension for the first 199 files and jpeg for the rest did break-up the image sequence loading though.

This is true (this breaks the pattern matching for sequentially named files), but this strategy won't let Avidemux mistake a JPEG image for an AVI container.

Quote from: butterw on February 02, 2021, 05:45:33 PMThe TinyPy crash with 200 files in a directory needs to be addressed

I didn't experience a crash with number of entries far exceeding 200 with the new code (well, on Linux), so I am cautiously optimistic that this problem doesn't exist anymore.

Quote from: butterw on February 02, 2021, 05:45:33 PMbut I'm not sure it is practical to edit videos with more than 200 segments in Avidemux

This is definitely both practical and a very valid use case, just not with 200 reference videos.
Title: Re: How to append 500 mp4 files into single video?
Post by: butterw on February 03, 2021, 12:00:07 PM
To confirm what as being said, I could load 500 jpg files without issue (using multiple get_folder_content batches) as long as the names were not in sequence and could create several thousands segments with them.

The filename pattern matching for image autoload works with a name sequence like this:
aaaa0300.jpg
aaaa0301.jpg
It stops if there is a missing digit or if there is a suffix:
aaaa0302b.jpg
aaaa0303.jpg
The only way to know how many images have been auto-loaded seems to be duration.