How to append 500 mp4 files into single video?

[butterw]

Using Win10 drag and drop, I'm crashing (exception filter error, crash.py) at about 230 jpg files. It also crashes, if I open less than 200 files, then attempt to append more.

So it seems currently Avidemux can't handle much more than 199 files.

 

[eumagga0x2a]

Did you test with hw accelerated video decode (DXVA2) enabled or disabled? Please disable and reduce editor cache size to minimum ( 8 pictures), restart Avidemux and retry.

[eumagga0x2a]

On Linux, with the undelying buildDirectoryContent modified to take a pointer to std::vector<std::string> as argument, I could successfully append 209 mp4 videos at 720p resolution.

But really, Avidemux consumes a ton of memory for all the editor caches and decoder instances, 5,5 GiB currently on my PC. At 1080p and with even more files, one would need probably a computer with 32 GiB of memory.

[eumagga0x2a]

Is it possible to raise the limit to 500 or 1000files or is this a TinyPy limitation ?

This is an arbitrary cap, the design of the function retrieving the content of a directory needed that. At the time as this code was written, this value was probably seen as more than sufficient.

The practical limitation is available memory as Avidemux creates a new decoder instance and a new editor cache for each reference video.

[butterw]

Using Win10 drag and drop, I'm crashing (exception filter error, crash.py) at about 230 jpg files. It also crashes, if I open less than 200 files, then attempt to append more.

So it seems currently Avidemux can't handle much more than 199 files.
 

This was with the default dxva2 HW-Accel enabled. But the result was the same with it disabled.

I could go higher by lowering the editor cache preference from 16 to 8 decoded pictures.

Memory usage was not high (80MB), but it still crashed.

[eumagga0x2a]

Images are a special case if their filenames are sequential. Were they random in your test?

Else drag'n'dropping N sequentially named .jpg files will load N + (N-1) + (N-2) + ... = N(N+1)/2 files, equal 20100 when N=200, easily hitting the operating system limit on open files.

[eumagga0x2a]

I've pushed a pretty massive patch which removes the cap.

[butterw]

Images are a special case if their filenames are sequential. Were they random in your test?

Else drag'n'dropping N sequentially named .jpg files will load N + (N-1) + (N-2) + ... = N(N+1)/2 files, equal 20100 when N=200, easily hitting the operating system limit on open files.

Image names were sequential and unknown to me Avidemux has a special built-in loader for image sequences which creates a single segment (! don't use drag-and-drop to open a long image sequence or only on the first image of the sequence). I tried remaming to .avi but got the same result.
 

[eumagga0x2a]

I tried remaming to .avi but got the same result.

The only one case where Avidemux relies on extension when loading files are VapourSynth scripts (vpy).

[butterw]

Using jpg extension for the first 199 files and jpeg for the rest did break-up the image sequence loading though.


I performed the test again this time with a single 10s mp4 file, which I appended over and over again until it crashed, which I can confirm occurs when it runs out of physical memory (VSWin64 v2.7.7 (210129_). For me, this occured before 300 files, but I'm running with 8MB of RAM. A 32GB machine should be able to open in excess of 1000 files this way (of course this also depends on the uncompressed size of the video).

If instead of trying to append all files sequentially, I save to file in Copy mode after loading 100 files, a much longer video sequence could be opened for the same amount of installed RAM. However, it might be simpler to just concatenate 100 files into 1 with ffmpeg, before loading in Avidemux.

The TinyPy crash with 200 files in a directory needs to be addressed, but I'm not sure it is practical to edit videos with more than 200 segments in Avidemux, there seems to be a lot of associated overhead.

[eumagga0x2a]

Using jpg extension for the first 199 files and jpeg for the rest did break-up the image sequence loading though.

This is true (this breaks the pattern matching for sequentially named files), but this strategy won't let Avidemux mistake a JPEG image for an AVI container.

The TinyPy crash with 200 files in a directory needs to be addressed

I didn't experience a crash with number of entries far exceeding 200 with the new code (well, on Linux), so I am cautiously optimistic that this problem doesn't exist anymore.

but I'm not sure it is practical to edit videos with more than 200 segments in Avidemux

This is definitely both practical and a very valid use case, just not with 200 reference videos.

[butterw]

To confirm what as being said, I could load 500 jpg files without issue (using multiple get_folder_content batches) as long as the names were not in sequence and could create several thousands segments with them.

The filename pattern matching for image autoload works with a name sequence like this:
aaaa0300.jpg
aaaa0301.jpg
It stops if there is a missing digit or if there is a suffix:
aaaa0302b.jpg
aaaa0303.jpg
The only way to know how many images have been auto-loaded seems to be duration.