"Too short" message on perfectly good files

[TCmullet]

I've grown to love a really slick encoding tool.  It's part of the Replay suite from Applian.  The tool is Replay Converter.  It's up to version 6 now.  In general, I'm using it with a slightly customized template which includes the "same quality" switch.  In my opinion, this program does a very fine job packing all possible quality into a video and still making it the smallest possible size.  Part of that ability is, I believe due to them setting large GOP sizes, which I've chaffed against in the past.  But now that I don't try to edit the resulting files, I don't mind.

What I mind is that sometimes a file I feed into it has the audio sync off a bit.  No problem, I just load it up into Avidemux and do some plus or minus shifting in ms.  That's not what I mind.  What I mind terribly is that probably 1/3 to 1/2 of the very finely encoded files I try to audio-shift in Avidemux, Avidemux barfs with this disgusting and really useless message, "Too short - The video has been saved but seems to be incomplete."  I have a gripe at two levels.

1.  I can find the GOP that was allegedly "bad", and can cut it out.  However, without cutting it out, the video will play fine in the timeline, both at normal speed and even single frame stepping.  I can see no flaw in any frame that RVC created.  I can (and have at times) cut out the one GOP just to save the rest of it AND accomplish my audio timeshift.  I can think of no excuse for such an otherwise fine program as Avidemux to EVER give this "Too short" message, at least in the context of a perfectly good file.  Do you say, "there's something flawed in it"?  Really then TELL ME WHAT THAT FLAW IS!!!  If you're going to abort a simple remuxing (and I've never seen other remuxers from my Mpeg2 days that EVER blew up), then you have a moral obligation to at least inform the user as to what the problem was!  I realize I'm being dogmatic here, but I believe I am in the right.  Back when I had a real life and career, I was a programmer and considered by many who worked with me to be a good one.  I learned you ALWAYS give some kind of meaningful information to the user when there's a problem.  ALWAYS.  Sometimes to not overwhelm them with tech-talk, you may have to give an error code and tell them to look it up or talk to customer support.  This "too short" message HAS to give information and it does not.  It's a serious flaw and no real excuse for it.  We sometimes work hard to create content and to have to throw it away because either the program can't handle "good data" and won't even tell us why, is inexcusable.

2.  The broader gripe I have, I have had for a long time.  I feel the ONLY time "too short" should EVER come out at us is when the storage space has run out.  There WILL be damaged files out there.  There just will be.  Granted, most of them will come from something other than one's own video camera.  But still.  A tool like this ought to ALWAYS go to the end.  If there are garbage chunks, the program should find a way to handle it, of course giving warnings about it to the user, along with relevant data items (for example, "Garbage data found at 1:03:39.027; dropping 13 frames") or perhaps giving an option, either in preferences or an interruption during the muxing itself, to replace lost frames with black.  I think most of the video streams have some kind of presentation time stamps that correlate video frames to audio packets.  This should be usable by Avidemux to find the next good data after a bad spot and resume normal muxing/copying.  I realize garbage detection and processing is not a pleasant thing to program for, but it is something that should be a high priority.

I did do searching for "too short" to see how this has been discussed before, but it appears to hardly been discussed at all.  Yet I cannot believe that there are not hundreds of people who would echo my concerns.  I do not have a particular propensity to acquired garbaged files (but I do catch a lot that is streamed). I'm in the middle of a crunch to get out a lot of files for a project I'm trying to do to stay afloat, and I need to not keep running into these inappropriate brick walls.  Every time I have to adjust audio sync, I dread that "too short" will appear in a perfectly good file, usually (but not always) encoded cleanly and excellently by Replay Converter.  What do I do now?  Do I release the video with the audio needlessly delayed by 167 ms, or do I compromise content by cutting out a 4-sec. GOP?  I hate both options.  Please consider addressing at least my 1st gripe.  It should be EASY to fix.  Admittedly #2 will be harder.

I cannot find a donate button, but if there was, and if I could FULLY trust Avidemux to "copy/copy" to the end ALWAYS (except for out of storage space), I'd make a donation.  It has become a VERY fine program, but this serious flaw greatly hurts its usefulness.  I frankly am very surprised it's gone this far and long without someone bring these things up.

[dosdan]

Well, if the program can't handle PTS issues in its current form, it should have 2 extras options:

1. Diagnose
2. Repair

If Diagnose found one or more issues it would offer to run a Repair pass. If you did not reply within 10s, (say as an option under Preferences), it would automatically run Repair for you. It would then write out a copy of the original, repackaged as MKV/MP4, if necessary, which it would then happily accept for A/B editing.

BTW, have you tried changing the timing offset using MKVToolNix GUI?

Dan.

[TCmullet]

This is MP4  I don't use MKVs.  But I do like your suggestions for correcting Avidemux.  It's a start.

[dosdan]

I presume you know the timing offset for an affected clip. Is it consistent for all files from this source? Or does it vary depending on how many "bad bits" are in the encode? 

If it's consistent, you could write a batchfile to run FFMPEG and just drag-and-drop an affected file onto the icon of say -300mS.bat.

Look at the batchfile here as a possible drag-and-drop template to run FFMPEG. Change the instances of .mkv to .mp4 so FFMPEG knows what container to use: http://avidemux.org/smif/index.php/topic,17390.msg78865.html#msg78865

Look here for a description of altering the timing offset using FFMPEG: https://wjwoodrow.wordpress.com/2013/02/04/correcting-for-audiovideo-sync-issues-with-the-ffmpeg-programs-itsoffset-switch/

[TCmullet]

I presume you know the timing offset for an affected clip. Is it consistent for all files from this source? Or does it vary depending on how many "bad bits" are in the encode? 

If it's consistent, you could write a batchfile to run FFMPEG and just drag-and-drop an affected file onto the icon of say -300mS.bat.

Yes, I have estimated (using playback of video with shifted audio in Avidemux which works well at that point) that my audio needs to come 133 ms sooner, therefore I set the delay to -133ms.  No it's neither.  This file as originally supplied to the RVC encoder was off by that much.  I don't think there are "bad bits".  But I'll still consider your bat file idea.  I can change the shift value with each file I need to shift.

But this is still a kludgy workaround of the severe and unjustified weakness of Avidemux as described.  (I say kludgy as I feel all bat file operations are kludgy at best, esp. when it's more than 1 or 2 parameters for a command, which I realize this case has to be.)  I didn't know that we could drag/drop onto an icon (shortcut) for a bat file.  Cool.

[Jan Gruuthuse]

>8>8 of the severe and unjustified weakness of Avidemux as described.  (I say kludgy as I feel all bat file operations are kludgy at best, esp. when it's more than 1 or 2 parameters for a command, which I realize this case has to be.)  >8 >8

A harsh comment for

Avidemux is a simple tool for simple video processing tasks. The keyword here is simple: it does not offer tools like a timeline, multitrack editing, you cannot freely move or splice audio and video clips from various sources. However, Avidemux allows you to do elementary things in a very straightforward way.

source: Avidemux Wiki

My user opinion of course.

[dosdan]

But this is still a kludgy workaround of the severe and unjustified weakness of Avidemux as described.  (I say kludgy as I feel all bat file operations are kludgy at best, esp. when it's more than 1 or 2 parameters for a command, which I realize this case has to be.)  I didn't know that we could drag/drop onto an icon (shortcut) for a bat file.

If the timing offset is consistent and the mapping is the same for all the affected files, then there is no real effort involved. You just drag-and-drop one clip or a whole bunch of them onto -133.bat, and then look in the "converted" sub-directory beneath where the original files were located. Alternatively, it would be trivial to leave the converted files in the same directory, but ending then in "_converted.MP4" to distinguish them for the original files. Or, if there was no higher errorlevel reported on an offset conversion pass (I think errorlevel 0 means "OK"), you could automatically delete the original.

Yes, in a perfect world ADM would be bullet-proof. But until perfection arrives, we either work with what we've got and work-around problems when we can, while hoping the developer(s) aren't burnt out yet and will get around to fixing it, (even if the problem is unlikely to affect them directly), when they can muster up the time and the motivation.


Dan.

[TCmullet]

If the timing offset is consistent and the mapping is the same for all the affected files, then there is no real effort involved. You just drag-and-drop one clip or a whole bunch of them onto -133.bat, and then look in the "converted" sub-directory beneath where the original files were located. Alternatively, it would be trivial to leave the converted files in the same directory, but ending then in "_converted.MP4" to distinguish them for the original files. Or, if there was no higher errorlevel reported on an offset conversion pass (I think errorlevel 0 means "OK"), you could automatically delete the original.

No, each file is uniquely "off" if at all.  Have to tune each one.  Guess I'll have to research how to install ffmpeg and hope installing it doesn't mess up something else I already have.  It's funny, I can remember the days I was averse to Windows at all, not wanting to give up my command line.  And now for years, probably due to pathing issues, I dislike anything not gui, and dislike messing with path commands.  But will consider your suggestions.

[TCmullet]

Avidemux is a simple tool for simple video processing tasks. The keyword here is simple: it does not offer tools like a timeline, multitrack editing, you cannot freely move or splice audio and video clips from various sources. However, Avidemux allows you to do elementary things in a very straightforward way.

source: Avidemux Wiki

My user opinion of course.

I'm not expecting "timeline, multitrack editing, ... from various sources, etc."  Yes, the simple elementary things only.  BUT, it's not unreasonable to expect those simple elementary things to work and to handle exception conditions, or at least report what the exception condition IS.   This is with regard to my 1st point...  I shouldn't be left to wonder for all eternity what ADM thought was wrong with that GOP.  It would be better knowing what the alleged error was even if not corrected by ADM, rather than not even being told what it was.

[TCmullet]

In many cases over the years, the "too short" message occurs on files way too big to send.  Often the whole point of "sending a sample" is thwarted by the very problem.  "I can't send a sample as the tool I have to extract a sample is failing to copy/copy the sample", is often the refrain.

However, in this case, the entire file is small enough (1.4 GB) that for a short time, I could upload it to and host it on my web server and ones here could download it to examine personally.  This is in case a programmer wants to see what's happening in this case to trigger the error.  If I was a programmer again, this is what I'd ask the one reporting an error.  Yes, I'd say, "please send me the file".

But I also think that the program should already be reporting whatever condition it CAN detect.  (Obviously, it's already detecting a condition and choosing to stop.  It's just not telling us what that condition is.)

[mean]

That error is usually the result of invalid timestamps
The muxer bails out
You have a message such as "DTS going backward" in the log or similar

[TCmullet]

I would like to see the errors.  I can then report them to the author of the conversion utility that I used.  But I have looked everywhere I can think of to find such a log.  Would you please tell me where the file is located?

[Jan Gruuthuse]

- Avidemux menu: Help
- Open the application data folder
- Attach compressed admlog.txt

Please reproduce the issue, open the application data folder from the "Help" menu in Avidemux, compress and attach the Avidemux log file (admlog.txt) to your reply.

[TCmullet]

- Avidemux menu: Help
- Open the application data folder
- Attach compressed admlog.txt

Thanks.  (Yes, I never woulda found it poking around in Windows.  Not unless I thought to do a programmed search for a folder named avidemux.)
Please see the attached zip file of admlog.txt.

If anyone wants to see it, I can upload the whole 1.4GB file (which looks to eye to be beautifully encoded, even when playing the trouble spot) to the host of my website.  Let me know.  It's only 1.4GB, which I can host there temporarily.  That small, yet holds 1 hr 8 min. of 720p60 sporting action.

[TCmullet]

I just ran into another encoded file with the symptom.  Fails very soon into the copy/copy.  I can't upload the video for this as it's over 3GB, but here attached is the log for IT.

But it seems I'm creating more files WITH this problem than without.  Oddly, it never seems to happen more than once throughout a 9-150 min. video.