hello,
i am using AviDeMux 2.7.4 (and AviDeMux 2.7.5 nightly) on windows 10. i recently recorded a big 16 hour chunk of TS file with my TV card.
now i wanted to split that 16 hour TS file into small separate video files of 90 minutes chunks.
but after AviDeMux created the indexfile i only can see and handle the very first 00:00:00.000 to 02:49:32.891 part of the total 16:04:55 of the TS file.
the VLC player is able to play back the whole 16:04:55 of the TS file without problems.
but AviDeMux only see the first 02:49:32 part.
that is what AviDeMux gives as properties:
=====================================================
Video
=====================================================
Codec 4CC: H264
Auflösung: 1280 x 720
Seitenverhältnis: 1:1 (1:1)
Bildrate: 50.000 fps
Gesamtdauer: 02:49:32,891
=====================================================
Zusätzliche Video-Eigenschaften
=====================================================
ExtraData-Länge: 00
=====================================================
Audio (4 active track(s))
=====================================================
Codec: MP2
Kanäle: Stereo
Bitrate: 32000 Bps / 256 kbps
Abtastrate: 48000 Hz
Gesamtdauer: 02:49:31,941
the 16:04:55 TS file has a file size of 48,7 GB (52.395.066.080 Bytes).
AviDeMux created an index file of 73,2 MB (76.758.523 Bytes).
any idea, how i can handle the whole TS file with AviDeMux?
Please move, rename or delete the idx2 file, then start the latest available Avidemux nightly, load the TS file so that Avidemux regenerates the index, close Avidemux, then compress (zip or 7z) and attach the admlog.txt file from %localappdata%\avidemux\ to your reply and provide the created idx2 file (it would be probably too large for an attachment even when compressed) via WeTransfer, Mega, Dropbox, Google Drive or an equivalent service to allow to assess the problem.
It could be a timestamp reset in the stream (which is not handled by Avidemux for MPEG-TS) or something completely different. The two files might give a hint.
here is the log file that AviDeMux created while opening and indexing that TS file:
see attachment
PS.: i used different versions of AviDeMux 2.7.5 nightly (win64 QT, win64, vc win64, win32) none of them could handle the whole TS file.
Thank you, will look at it in a moment.
By the way, is the report https://sourceforge.net/p/avidemux/bugs/98/ yours? I don't have a SourceForge account and I'm not going to create one, but reports there are read, of course, just no action can be taken upon them if an inquiry is necessary.
Quote from: beta-tester on May 10, 2020, 08:47:49 AM
PS.: i used different versions of AviDeMux 2.7.5 nightly (win64 QT, win64, vc win64, win32) none of them could handle the whole TS file.
Sure, the MpegTS demuxer code is the same.
yes, thats my report...
here the link to the uploaded 7z'ipped index file:
http://www.filedropper.com/diverse202001011355tsidx2
i couldnt find another file hoste without registration.
from the indexing dialog of ADM i could see that the last number of processed frames was aroud 2891700 frames.
calculating with 50fps content of the TS file that looks like the whole 16 houres.
so i think the index file should contain the complete 16 hours as well (hopefully)
Thank you, got the file. NB: WeTransfer requires neither registration nor an email address (and doesn't ask users to solve captcha).
Already the log provides some hints:
[tsDemux] Found 2894693 video frames
At 50 fps this means ~ 16 hours of video, which matches the expectations.
[addReferenceVideo] 08:32:44-036 min increment 00:00:00,020 = 20000 us for frame 1
[addReferenceVideo] 08:32:44-036 max increment 339:53:08,480 = 18446744025987712772 us for frame 2386055
Looks very much like timestamp reset (so that the code which doesn't expect decode timestamps going back results in unsigned values wrapping around) close to the end. The duration we get is the duration of the part of the stream after the presumable timestamp reset.
It is a bit more complicated. The duration is the last presentation timestamp - the first presentation timestamp, with timestamp reset at offset 39436043720 Bytes (~36,73 GiB).
If you have a Linux system or a tool equivalent to head or dd at hand, you could just copy the first 36,72 GiB of the stream which Avidemux should handle without problems. The same applies to the tail of the file from the offset 39436043720 on, it is just the boundary which is currently not handled.
do you think the issue is fixable, maybe by changing the datatype of that offset/index?
It would be great if you could provide a small part of the original stream containing the timestamp reset as a sample, e.g. by using
dd if=diverse_20200101_1355.ts of=mpegts-tsreset.ts bs=1M skip=37500 count=200
The file "mpegts-tsreset.ts", 200 MiB in size, would contain the boundary.
Quote from: eumagga0x2a on May 10, 2020, 09:33:34 AM
If you have a Linux system or a tool equivalent to head or dd at hand, you could just copy the first 36,72 GiB of the stream which Avidemux should handle without problems. The same applies to the tail of the file from the offset 39436043720 on, it is just the boundary which is currently not handled.
yes, that is a good idea to split the TS file in chunks that ADM can handle.
can i split TS files to any size or is there a alignments i have to respect?
Quote from: beta-tester on May 10, 2020, 09:41:33 AM
do you think the issue is fixable, maybe by changing the datatype of that offset/index?
The issue might be fixable in the code of the MpegTS demuxer (probably not in time for 2.7.6, but who knows). It could be partially worked around as described above by splitting the input file.
Quote from: beta-tester on May 10, 2020, 09:45:23 AM
Quote from: eumagga0x2a on May 10, 2020, 09:33:34 AM
If you have a Linux system or a tool equivalent to head or dd at hand, you could just copy the first 36,72 GiB of the stream which Avidemux should handle without problems. The same applies to the tail of the file from the offset 39436043720 on, it is just the boundary which is currently not handled.
can i split TS files to any size or is there a alignments i have to respect?
Any size, just the first GOP after the cut will be probably lost.
I don't think splitting input file using Linux on command line is a viable approach for the majority of users, so finding a code fix would be preferable by far. Only possible when a sample is available 8)
Thank you very much, got the file with the reset happening at offset ~ 109 MiB. The problem is that Avidemux happens to load it just fine. Need to understand why it works here, will look into it a bit later.
i now splitted the huge TS file in three overlapping smaller ones that ADM can handle.
so i do not loose anything.
thank you for the hint with splitting the file as workaround.
The funny thing is that timestamps wrapping around when an unsigned 32 bits integer is full like in this case is well handled in the demuxer code as long as it happens only once (i.e. after at most 0xFFFFFFFF MPEG ticks / 90000 Hz ≈ 47721,859 s ≈ 13 h 15 min). In theory, recodings up to 26 h 30 min duration should be fine. The problem is definitely related to wrap handling, but not that easy to spot.
By chance, do you have a real (not live) Linux system with write access to the location where the original huge TS file is stored to allow easy compiling of patched Avidemux versions?
By all means, please keep the original huge file to allow troubleshooting and verification in the future.
Quote from: eumagga0x2a on May 10, 2020, 03:19:29 PM
By all means, please keep the original huge file to allow troubleshooting and verification in the future.
sorry i saw your post too late. i deleted the huge file already to free diskspace - i can't restore tha file anymore.
i can recorde a new one from the same tv channel with the similar duration and try to reproduce the issue...
i will tell you tomorrow if the new recorded 16:05 hours TS file will show the same issue.
i am optimistic that i can reproduce that issue...
i think i remember that it is not the first time that i could not to handle that huge TS files in the past.
Quote from: eumagga0x2a on May 10, 2020, 03:19:29 PM
By chance, do you have a real (not live) Linux system with write access to the location where the original huge TS file is stored to allow easy compiling of patched Avidemux versions?
i have the possibility to run real linux (ubuntu 20.04 LTS amd64) on the PC.
Quote from: beta-tester on May 10, 2020, 05:36:12 PM
Quote from: eumagga0x2a on May 10, 2020, 03:19:29 PM
By all means, please keep the original huge file to allow troubleshooting and verification in the future.
sorry i saw your post too late. i deleted the huge file already to free diskspace - i can't restore tha file anymore.
That's a pity.
Quotei can recorde a new one from the same tv channel with the similar duration and try to reproduce the issue...
i will tell you tomorrow if the new recorded 16:05 hours TS file will show the same issue.
If you have already started to record (DVB-C, I assume?), then please go on. Else I'll try it later on my own computer, with the benefit of direct access to a testcase if I manage to reproduce the problem.
Quotei have the possibility to run real linux (ubuntu 20.04 LTS amd64) on the PC.
Great, if you happen to get hold of a sample suitable to reproduce the problem by tomorrow, I'll post patches here to add debug messages which I hope would help to analyse the issue and guide you through the trivial process of building Avidemux on Linux.
ok, recording 16:05 TS file finished, same issue with ADM and that TS file on windows (nightly 2.7.5 20200504) and on linux (2.7.4).
i copied the TS file to a virtual box machine where ubuntu 20.04 LTS is installed.
there i compiled avidemux 2.7.4 from source - so the build environment is prepared.
i also installed TeamViewer in case you need remote access to that machine.
if you want, i will send you the TeamViewer ID and the password to remote access the machine.
but i can't keep that PC with the virtual machine turned on permanently...
if i should do testing, please tell me from where i have to download the prepared nighly source code of ADM.
Great, thanks! Here we go. I assume git is already installed. To build Avidemux from the current git, do
git clone https://github.com/mean00/avidemux2.git
cd avidemux2
git submodule update --init --recursive
If you have already all build deps, skip
bash createDebFromSourceUbuntu.bash --deps-only
We are not going to use any packaging, therefore the next step is
bash bootStrap.bash
If everything went well, copy "run_avidemux_template.sh" as "avidemux" (or any other name you like to use to launch Avidemux) to a directory in your $PATH and make it executable. You should be now able to run Avidemux by executing
avidemux
in the terminal. Once all that works, I'll attach patches here to produce diagnostic messages hopefully helpful during troubleshooting. I'd like to avoid pushing them to the repository.
Would you please provide the new *.idx2 file?
Wrap handling is probably just the red herring. This was from the old admlog.txt:
[addReferenceVideo] 08:32:43-962 Original frame increment 00:00:00,020 = 20000 us
[addReferenceVideo] 08:32:44-036 min increment 00:00:00,020 = 20000 us for frame 1
[addReferenceVideo] 08:32:44-036 max increment 339:53:08,480 = 18446744025987712772 us for frame 2386055
[addReferenceVideo] 08:32:44-036 About 20000 microseconds per frame, 2894693 frames
[getVideoDuration] 08:32:44-036 Found maxPts =02:49:32,811, 3 frames from the end
[getVideoDuration] 08:32:44-037 Found maxDts =02:49:32,711, 2 frames from the end
[getVideoDuration] 08:32:44-037 Using PTS..
[getVideoDuration] 08:32:44-037 Using duration of 02:49:32,871
[addReferenceVideo] 08:32:44-037 The first frame DTS = 770 ms
[addReferenceVideo] 08:32:44-037 The first frame has a PTS > 0, adjusting to 850 ms
[updateStartTime] 08:32:44-037 Using pts2=00:00:00,850 to get dts, as this video does not start at zero
[ADM_verifyDts] 08:32:44-038 Verifying DTS....
[ADM_verifyDts] 08:32:44-038 Checking from 1 to 2894693
And now:
[addReferenceVideo] 10:24:26-490 Original frame increment 00:00:00,020 = 20000 us
[addReferenceVideo] 10:24:26-551 min increment 00:00:00,020 = 20000 us for frame 1
[addReferenceVideo] 10:24:26-551 max increment 339:53:08,480 = 18446744025987712772 us for frame 2386048
[addReferenceVideo] 10:24:26-551 About 20000 microseconds per frame, 2894766 frames
[getVideoDuration] 10:24:26-552 Found maxPts =02:49:34,401, 1 frames from the end
[getVideoDuration] 10:24:26-552 Found maxDts =02:49:34,341, 0 frames from the end
[getVideoDuration] 10:24:26-552 Using PTS..
[getVideoDuration] 10:24:26-552 Using duration of 02:49:34,421
[addReferenceVideo] 10:24:26-552 The first frame DTS = 900 ms
[addReferenceVideo] 10:24:26-552 The first frame has a PTS > 0, adjusting to 980 ms
[updateStartTime] 10:24:26-553 Using pts2=00:00:00,980 to get dts, as this video does not start at zero
[ADM_verifyDts] 10:24:26-553 Verifying DTS....
[ADM_verifyDts] 10:24:26-553 Checking from 1 to 2894766
Exactly the same bogus max increment value at almost identical frame number – this time no timestamp wrap even close to that spot.
Could be a rounding error in timeConvert(), however, I'd expect that 2.7.4. behaves differently here.
do i understand it right, the new recorded TS file have fewer wrapping issues than the old one?
does it mean the stream in the TS file is kind of broken?
i sent you the index file and the log from the console from the linux version now.
Thank you, I've got the files from a Linux build.
No, the stream is not broken. It is just that my first idea about the location in code which might be responsible was wrong.
edit: Avidemux can't handle more than one timestamp wrap in a stream, but that is a different problem.
Could you please open the script console in Avidemux on Linux while the new huge TS file is loaded, and run the following commands?
ed = Editor()
for frame in range(2386000,2386100): ed.printTiming(frame)
The output is printed to the terminal (there should not be any errors), please post it.
here the output...
Thank you, so timing is wrapping around at UINT_MAX in MPEG-TS ticks offset from the start (not absolute value):
QuoteFrame 2386039 Flags 0000 (P/F) DTS 13:15:21,680 PTS 13:15:21,780
Frame 2386040 Flags 4000 (B/F) DTS 13:15:21,700 PTS 13:15:21,740
Frame 2386041 Flags 4000 (B/F) DTS xx:xx:xx,xxx PTS 13:15:21,720
Frame 2386042 Flags 4000 (B/F) DTS 13:15:21,740 PTS 13:15:21,760
Frame 2386043 Flags 0000 (P/F) DTS 13:15:21,760 PTS 00:00:00,001
Frame 2386044 Flags 4000 (B/F) DTS 13:15:21,780 PTS 13:15:21,820
Frame 2386045 Flags 4000 (B/F) DTS xx:xx:xx,xxx PTS 13:15:21,800
Frame 2386046 Flags 4000 (B/F) DTS 13:15:21,820 PTS 13:15:21,840
Frame 2386047 Flags 0000 (P/F) DTS 13:15:21,840 PTS 00:00:00,081
Frame 2386048 Flags 4000 (B/F) DTS 00:00:00,001 PTS 00:00:00,041
Frame 2386049 Flags 4000 (B/F) DTS xx:xx:xx,xxx PTS 00:00:00,021
Frame 2386050 Flags 4000 (B/F) DTS 00:00:00,041 PTS 00:00:00,061
Frame 2386051 Flags 0000 (P/F) DTS 00:00:00,061 PTS 00:00:00,161
Frame 2386052 Flags 4000 (B/F) DTS 00:00:00,081 PTS 00:00:00,121
Could you please test whether the attached patch fixes the problem? You don't need to re-index the TS file.
In the avidemux2 directory (i.e. in the top source directory)
patch -p1 < /path/to/demuxer-mpegts-try-to-fix-wrap.patch
(it should apply cleanly), then (1)
bash bootStrap.bash --rebuild --without-core --without-qt --without-cli
To reset changes made by the patch, you can use
git checkout avidemux_plugins/ADM_demuxers/MpegTS/ADM_tsComputeTimeStamp.cpp
in the top source directory followed by (1).
looks loke a deadlock...
one CPU core is at 100%
nothing happens.
i killed the process once. (p1.log)
after that i tried with re indexing the TS file; the indexing process finished successfully but then again ADM hangs at the same situation.
EDIT: ubuntu is now complaining, that avmdemux3_pt5 is not responding [force quit][wait]
Oops, thank you, next try (you need to adjust the patch command accordingly).
i created a "synthetic" TS file of 16h duration by using ffmpeg.
it's compressed size is only 16MB and unzipped it is 1GB.
so i can upload it easely and it does not contain potentially copyright protected content.
https://www.file-upload.net/download-14071227/out-test-files.7z.html
created with (contains only video data):
ffmpeg -framerate 50 -loop 1 -i ~/Pictures/test_1280x720.jpg -t:v 57600 -vf scale=1280:720 -r 50 -pix_fmt yuv420p -c:v libx264 out.ts
created with (contains video and audio data):
ffmpeg -framerate 50 -loop 1 -i ~/Pictures/test_1280x720.jpg -stream_loop -1 -i ~/Music/test_1kHz_30s.wav -t:v 57600 -vf scale=1280:720 -r 50 -pix_fmt yuv420p -c:a mp2 -b:a 256k -ar 48000 -c:v libx264 out2.ts
it shows up the same issue, under linux and windows.
i did not tried your new patch yet, but i will try it in a moment.
ok, i tried your second patch...
it opends the real TS file and the synthetic one without problems. i can see the whole 16h of video and i can move the slider to any position. that is nice.
BUT playback the TS file in ADM is problematic.
i can play back both TS files from position 00:00:00.000.
but i can't play back the real TS file from position above 14:00:00.000 ... then ADM gets stuck (not responding) and produces in the background many many MBytes of endless log output.
...
[Composer::getPCMPacket] Track 0, 0x55b4d09c2030 : drift -1530441629, computed : 13:57:33,660 got 00:04:04,494
[Composer::getPCMPacket] Track 0:55b4d09c2030 : Dropping packet 50253660 last =244494
...
i had to cut the log file from about 500MB down to 1MB so send it here.
see attachment.
but the strange thing is, that the synthetic TS file out.ts can be played back from any position. but that file does not contain any audio stream.
and the message "[Composer::getPCMPacket] ..." looks like an audio stream issue here.
i have to generate a synthetic out.ts file with an audio stream first to see if the issue can be reproduced with a synthetic TS file as well.
but rendering a TS file with ffmpeg will take on my PC about 7h without audio...
Thank you very much, there must be a similar wrap problem in the generation of audio seek points, not tackled by the patch yet. Will look into it very soon.
Could you please test whether the attached patch helps with audio timestamps wrapping around?
To discard changes from this patch, you can use
git checkout avidemux_plugins/ADM_demuxers/MpegTS/ADM_ts.h avidemux_plugins/ADM_demuxers/MpegTS/ADM_tsAudio.cpp
YES, that looks great now... thank you very much!!!
i tried to copy video and audio to a mkv file successfully from around 13:xx:xx.xxx to around 14:xx:xx.xxx.
i am currently try to convert video and audio, but that will take a longer while...
here the log of the current process...
(including log of the copy to mkv, a convert to x265/aac i canceled and convert to mpeg2/mp3 that is still on progress)
Glad to hear that the patch fixes the issue. If you have sufficient storage space, could you please test whether it still works with a video twice as long (i.e. longer than 27 hours)? The steps would be as following:
1. load the recorded stream in Avidemux
2. append it to itself
3. save in copy mode as MPEG-TS
4. load the result
Then please test whether seeking and starting playback works all the way to the end of the entire video. Thanks!
you made a great job...
all tests passed.
- appending 16h + 16h TS to 32h TS file - ok
- seeking with all possible knobs (frame by frame, I-frame by i-frame, seek wheel, ...)- ok
- play the video in ADM from several positions (0h, ~14h, ~17h, ~29h, ...)- ok
- saving the 32h TS file - ok
- deleting portions from the whole the 32h video - ok (
a message popped up - see screenshot) - reopening the saved 32h TS file - ok
- seeking with all possible knows (frame by frame, I-frame by i-frame, seek wheel, ...)- ok
- play the video in ADM from several positions (0h, ~14h, ~17h, ~29h, ...)- ok
- play the video in ADM from the transition part (16:01h, 16:06h)- ok
- saving a part from the transition from one 16h video to the appended 16h video of the 32h TS file - ok
EDIT: stupid me. forget the screenshot, i forgot to align the a- and b- markers to an i-frame
Great, thank you very much for your help. The fix wouldn't be possible without it. I'll commit and push the changes once I've finished my current work on editor.
can i delete all my huge test files now or shall i keep them for a while...
better i ask you, befor i do silly things again ::)
You would do me a great favor if you keep the 16 hours long recording for future testing (the double duration sample can be easily recreated if necessary). At least until a month or two past the 2.7.6 release, please.
is this TS file (see attached log) more interesting to you than the previous one?
if so, i would delete the previous TS file and keep the current one.
i have to delete one of the two big TS files, i need the disk space.
Is the indicated duration of 17:49:54,673 correct (I assume your Avidemux build is patched as otherwise it would be too old to handle it)? If yes, please keep the previous recording if somehow possible because its duration exceeds ~27 hours, i.e. it allowed to verify correct handling of the second "a uint32_t would now wrap around" event, not just the first one.
By the way, why haven't you updated to the current git master? There is some new stuff there in dire need for real-world testing including but not limited to an updated internal FFmpeg.
yes, the length of the TS file is 17:49...
but the previous TS file was 16h... only the "doubled appended" TS file was twice as long.
why i didn't updated my current git master?... because i am too lasy... i didn't wanted to apply the patches and recompile everything again... :-[
EDIT: i thought i have to apply the patches... now i see i don't.
Quote from: beta-tester on June 02, 2020, 10:14:53 AM
only the "doubled appended" TS file was twice as long.
And it is now gone? In this case it is up to you which one to keep.
Quotei didn't wanted to apply the patches and recompile everything again... :-[
The fix for MPEG-TS timestamps wrapping around was committed and pushed to git master three weeks ago, all official nightly builds uploaded after that date should handle MPEG-TS streams longer than 13 hrs 16 min just fine.
Please note to pull from the git master successfully you would need to reset the files modified by the patch as explained in https://avidemux.org/smif/index.php/topic,19110.msg88908.html#msg88908 (https://avidemux.org/smif/index.php/topic,19110.msg88908.html#msg88908) first.
Quote from: eumagga0x2a on June 02, 2020, 10:50:28 AM
Quote from: beta-tester on June 02, 2020, 10:14:53 AM
only the "doubled appended" TS file was twice as long.
And it is now gone? In this case it is up to you which one to keep.
i thought i am able to re-create the twice as long file when necessary - and will do, when you ask for.
i recompiled avidemux...
but i think i will use the precompiled nightly builds netx time,
i didn't realized before, that the changes were committed by you for the "wrap" issue.
(i feel i get more and more stupid the longer i can't go to the office to work ::) )
thank you.