Crashes when appending many images together, via Drag&Drop

[eumagga0x2a]

Drag&drop is identical with performing the append action in the "File" menu, this is so by design and is not going to change without a very valid reason. When a file is passed to the code which probes its content, it is too late to decide that we actually wanted to treat it as a frame of a single Motion JPEG video, auto-appended in the "Pictures" demuxer.

Limiting auto-append to sequentially named image files allows to pass a single bounds-checked file name to the demuxer which then can deduce all file names by incrementing a counter. From the point of view of Avidemux, it has loaded just a single file, all the complexity of handling individual images encapsulated and entirely concealed by the demuxer.

However, it doesn't mean that we should waste memory in scenarios with images appended in the editor, [editor] Reduce editor cache size for ultra-short ref videos like images to save memory should substantially reduce memory usage here (the cosmetical part of the changeset unfortunately forgotten in the commit message).

Additionally, a bug in editor (seek to the end broken in a video consisting of individial images appended in the editor) was found and fixed, thank you for your input.

[dosdan]

Example "20191107-154534-214.jpg"
The left 8 digits represents the date = YYYYMMDD
The middle 6 digits represent a time of day = hhmmss, using a 24 hour clock.
The right 3 digits are additional milliseconds.
All fields are padded with zeros, so the full timestamp is always 19 characters long, not including the filename extension (.jpg, .png, .tif, or .bmp).
So the above example represents 2019 Nov 7, at 15h45m34.214s

There's no problem with this sorting correctly under Windows, by name, on an NTFS-formatted drive. Quoting from https://ss64.com/nt/dir.html

Sorting
The default Sort Order, if you don't specify anything with /O, on an NTFS drive will be in sort-of-alphabetical order or on a FAT USB thumb drive, then the order will be based on the order in which files were created and deleted and the lengths of their names.

So, now we come to the problem of the image names not being sequential. Using FFMPEG, there is no need to copy or sequentially rename/renumber these files, as FFMPEG can accept a list of filenames. So the first step in this exercise is to produce a set of files that have their frames showing sequential numbers. This will be used to make a set of test JPGS so you can confirm that the correct sort order is used and that no frames are missing: each image shows one frame-number.



Note: I reduced the dimensions of this screen capture to save display space in the forum, so it looks more jagged here than it really is.

If distributed as an un-compressed AVI, this clip would be enormous. But as an MP4 it's only 2.5MB. Here is a 1000-frame MP4, with its frames numbered from 0000-0999, available for download:

https://dl.dropbox.com/s/q6s9v1ea4cgcr6g/1000_sequence.mp4

While you could use this file to create 1,000 separate image JPGs, which is what I did for much of setting this up, I think 250 frames, which is 10s at 25fps, will be a better choice for when we later change the duration of the sequence, as it makes the mental arithmetic easier. I'd suggest creating a C:\Test directory, copying FFPMEG.EXE to this directory if it's not in your PATH list and, from there, using the md extracted command to make a C:\Test\extracted sub-directory for all the test images.  All our actions will performed from the directory a level up from extracted.

To extract the first 250 images from 1000_sequence.mp4 use:

ffmpeg -i 1000_sequence.mp4  -qscale:v 2 -frames:v 250 extracted\output_%03d.jpg

This will produce this sequence of image files: 20191107-154001-214.jpg to 20191107-154250-214.jpg in the extracted sub-directory. Each is 1680x1050. It doesn't matter here that this is not correct HHMMSS naming. The important thing for this test is that the filename ending ("-214") is not varying sequentially. This will confuse programs that look at the end of the filename to determine the suitability for sequential loading.
-qscale:v 2 controls the quality of the the JPGs. "2" is fine.
-frames:v 250 limits the extraction to the first 250 frames. Leave it out if you want the full 1000 frames.

Now, FFMPEG is flexible enough to auto-load this sequence:

ffmpeg -i extracted\20191107-154%03d-214.jpg -c:v copy test.mkv

But this wouldn't work if the naming format was actually HHMMSS. For that we need to read from a list of name-ordered filenames. This list can also be used to alter the duration each image will be displayed.

Create_list.bat

Code Select Expand

@echo off
SETLOCAL EnableDelayedExpansion
if exist mylist.txt del mylist.txt
for %%I in ("extracted\*.jpg") do (echo file '%%~fI' >> mylist.txt)   
start /i mylist.txt


The start of mylist.txt looks like:

file 'C:\Test\extracted\20191107-154001-214.jpg'
file 'C:\Test\extracted\20191107-154002-214.jpg'
file 'C:\Test\extracted\20191107-154003-214.jpg'

The concat demuxer in FFMPEG wants each entry in this list to start with "file".
'%%~fI' produces the full pathname. If I had used just "%%I" it would have produced the relative path of

file 'extracted\20191107-154001-214.jpg'

While that works, it's probably safer to use the full pathname.

To assemble the MKV (which will be in C:\Test) use:

ffmpeg -f concat -safe 0 -i mylist.txt  -hide_banner -y -c:v copy output.mkv
-f concat forces FFMPEG to use Concat demuxer, which can read from a filelist, for input.
-safe 0, before specifying the name of the input list, stops the demuxer getting upset over the presence of hyphens in the filename (valid under Windows).

Output.mkv is 10s, 25fps and 17.8MB. If your media player has a blank screen, (probably won't be an issue with MJPEG), include the -pix_fmt yuv420p option.

In actual use, you'd create the list and then use it to concatenate the file and wrap it up a MKV, using just one batchfile, rather than the two, separate operations I've showing here. If needed, you could put a pause command between these two sections of the batchfile, allowing you to delete any untanted lines in mylist.txt in Notepad, before saving the changes, closing Notepad and pressing Enter to continue on to the concatenation stage.

Say the time-lapse action is too fast or too slow in the output file. The file list can include 2-lines per image, with the 2nd line specifying the image duration in secs. To add this feature, change the main line in Create_list.bat to:

for %%I in ("extracted\*.jpg") do (echo file '%%~fI' >> mylist.txt & echo duration 0.04 >> mylist.txt)

The start of the list then looks like

file 'C:\Test\extracted\20191107-154001-214.jpg' 
duration 0.04
file 'C:\Test\extracted\20191107-154002-214.jpg' 
duration 0.04 

An image duration of 0.04s is the duration of one frame at 25fps (40ms). Increasing this will start replicating frames, slowing down playback, while reducing it will drop some image frames, speeding up playback. You can alter the image duration value in the batch line above to suit yourself. 

An example: setting the duration to 0.2 changes the overall duration of the output clip from 10s -> almost 50s. And MediaInfo shows a change in framerate from 25fps -> 5fps.  It also changes the number of frames in the clip from 250 -> 1245. Due to a minor bug in the Concat demuxer, it drops the last image (here, 5 frames) if you specify duration. The work-around for this is mentioned in https://trac.ffmpeg.org/wiki/Slideshow

(Due to a quirk, the last image has to be specified twice - the 2nd time without any duration directive)

But the small difference in overall duration is usually not worth worrying about.

Despite the reported increase in frames with this playback slow-down, (I checked this out in VirtualDub), the output file is still 17.8MB.

Dan.

[JulianDude]

I want to thank everyone who has helped find a workable solution to my time-lapse image appending issue.  I particularly appreciate all the help dosdan and eumagga0x2a provided.  Since my image filenames are based on a timestamp, to simplify my workflow, I have chosen not to rename them as a sequence of numbered frames. Based on using ffmpeg.exe, I created a slightly modified batch files to what dosdan suggested.  I've attached a copy.  I'm also accessing ffmpeg via another small batch file so I didn't need to add ffmpeg's path to an environment path variable.

I did notice that the image filenames can't contain any single quote marks.  I tried changing dosdan's suggested code to use double quote marks around the paths in the created list file, so the names could then contain single quotes.  But apparently ffmpeg didn't like that.  Maybe there is a solution, but I gave up.

Also, my batch file needs to be placed in same directory as the image files.  I experimented with dropping a folder of images onto my batch file icon, which I'd prefer. And I got that to work using "FOR /R", but had to eliminate the directory file sort feature (which I feel is necessary), and I didn't know how to limit that /R solution to avoid potential sub-folders. I'm not a batch file guru like some of you guys.

BTW, I am happy with this solution, but will look forward to any image drag&drop improvements that may be made to ADM.  Thx again.

[dosdan]

JulianDude, here's part of your batchfile:

Code Select Expand

call ffmpeg -f concat -safe 0 -i _appendList.txt  -hide_banner -y -c:v copy _Output.mkv
del _appendList.txt

REM -------------------------------------------------------------------------------
REM NOTE: This batch runs ffmpeg via another small batch file, called ffmpeg.bat. The ffmpeg.bat file contains a REM path to where you actually store ffmpeg.exe. ffmpeg.bat should be placed in a folder that is already on Windows REM search path. That way, ffmpeg.exe can be found without needing to include a full path to it in a cmdline or batch REM file, nor needing to add another path to Window's environment variables.

REM The good place to put ffmpeg.bat is at:
REM "%USERPROFILE%\AppData\Local\Microsoft\WindowsApps"

REM ------- Here is an example ffmpeg.bat (don't include the "REM's"). Edit for your ffmpeg path  ------
REM @echo off
REM "C:\PortableEXEs\_Portable EXEs\ffmpeg-20191108-e700038-win64-static\bin\ffmpeg.exe"  %*


I think this is a messy solution: calling a batchfile named FFMPEG.BAT just because FFMPEG.EXE is not in the current directory.  You now need to have FFMPEG.BAT in either your current directory or in a pathed directory. 

I also don't like the idea of including the version name of a commonly-used (for the video fraternity, at least) utility. It's an OK practice if you have a program that uninstalls itself when upgrading, as it will remove the old program directory. Otherwise you end up with multiple directories for multiple upgrades of a program. Utility programs usually are stand-alone programs without installation programs, so it's a straight copy-to-install.

Instead, I'd suggest having a pathed directory e.g. C:\utils or C:\utilities into which you put all these types of program. You can also put all your batchfiles here. (If you text editor remembers your last used directory or keeps reloading one or more text files you have recently been working on in previous sessions and haven't yet closed, e.g. Notepad++, every time you go to create or edit a batchfile it will be looking in the correct place. (Alternatively, have another pathed directory, C:\batchfiles.)

In my case, since I regularly build Win64 versions of FFMPEG (https://github.com/m-ab-s/media-autobuild_suite), I have both C:\utils (I don't keep FFMPEG.EXE & FFPROBE.EXE here) and C:\Mingw\media-autobuild_suite-master\local64\bin-video (which is where the compiled video programs end up) included in my PATH list.

I'll look into single inverted commas and FFMPEG. Perhaps it's just a Concat demux input restriction?



Now, consider portable operation. You could carry both ffmpeg.exe &  _Append all Images.bat around on a USB stick. Let's call this USB's drive "X:". We're currently in a sub-directory on C:, say "C:\test directory\extracted". From here we issue:

"X: _Append all Images" produced by typing x:_  or x:_a and then pressing the Tab key to find a filename match on X:, pressing Tab again if it was an incorrect match.

This would have an problem if FFMPEG.EXE isn't in the PATH list.  However, assuming that FFMPEG.EXE & _Append all Images.bat will be found in the same location you can use a variant of the %0 parameter. This will only operate inside a batchfile.
Trying this test batchfile while we are on C: and the batchfile is on X:

Find starting loc.bat

Code Select Expand

@echo off
cls
echo Batchfile name and loc used on command line:  %0
echo Batchfile drive letter:  %~d0
echo Batchfile directory:  "%~p0"
echo Batchfile full path:  "%~dp0"
echo Full pathname to be used to run FFMPEG:  "%~dp0ffmpeg.exe"

Running this from C: with the batchfile on X:
"x:Find starting loc.bat"
Result:
Batchfile name and loc used on command line:  "x:Find starting loc.bat"
Batchfile drive letter:  X:
Batchfile directory:  "\"
Batchfile full path:  "X:\"
Full pathname to be used to run FFMPEG:  "X:\ffmpeg.exe"

And assuming we were in the 2nd level extract sub-directory (c:\test directory\extracted), with the program and batch files situated in the directory above this (c:\test directory), we can use  "..\" (directory above this one) to run the batchfile from where we are.

"..\find starting loc.bat"
Result:
Batchfile name and loc used on command line:  "..\Find starting loc.bat"
Batchfile drive letter:  C:
Batchfile directory:  "\test directory\"
Batchfile full path:  "C:\test directory\"
Full pathname to be used to run FFMPEG:  "C:\test directory\ffmpeg.exe"


So, if you are prepared to keep ffmpeg.exe and  _Append all Images.bat together, and they're not in the PATH list, I suggest changing the start of your FFMPEG line inside your batchfile from

call ffmpeg

to

"%~dp0ffmpeg.exe"

Dan.