Bugs as reflected in how AviDemux responds to Hauppauge Colossus

[TCmullet]

I think these two bugs are related, but in either case, I hope they can be fixed.  These are both SD captures, not HD.

This first file (contained in a zip file) is 15 seconds.  It's very odd that AviDemux shows that there's only 1 keyframe, the first frame.  Yet it behaves like there are 29 key-frames, as evidenced that you can jump to all 29 key-frames.  But only the first one shows that it's an I-frame.  All the others show as a P-frame.  Obviously just a mistake in how the program SHOWS what kind of frame.  We really need it to correctly show that it's an I-frame, not a P-frame.
http://www.tomsgoodfiles.com/2013-2-20-22-38-17.sample-SD3-clip-from-Tom-Cook.zip

I think the 2nd file has the same problem (I believe there are multiple key-frames) but behaves differently.  AviDemux crashes.  The capture begins while viewing my DVR menu of shows to play.  I start recording, then pick the show and play it.  Total is also about 15 seconds.  The Colossus successfully records situations like this.  Maybe there's something funky about what happens when the DVR switches from showing its menu to showing the show.  It's probably briefly changing what format of audio is recorded.  At the very least, we need to be able to have AviDemux chop off the menu stuff at the beginning, by being able to skip to later keyframes.  But in this case, it blows up.  In other cases, it has not blown up, but it refuses to see any more key-frames (labelled as "P") and jumps to the very end of the file.  (I can't share those, as they are way too big to upload anywhere.  I will try to create a small one.)  The menu is very useful to record in the raw captured file, as it contains info about the clip, so I hope we won't be told "don't capture that".  Here's the 2nd file.
http://www.tomsgoodfiles.com/2013-2-20-22-43-10.sample-SD4-clip-from-Tom-Cook.zip

[TCmullet]

I was able to create another file with the DVR "shows menu" captured at the start and AviDemux did NOT blow up.
http://www.tomsgoodfiles.com/2013-2-20-23-42-4.sample-SD5-clip-from-Tom-Cook.zip
It's a minute long, has 2 I-frames (at least that AviDemux can see and jump to), and plays through the whole file nicely, just as the other 2 do.  But there's only those 2 I-frames it can jump to.  Yet, I have to believe there are many key-frames throughout the entire 60 seconds.  Need to be able to get to them in a file such as this.

Again, if AviDemux could get to those keyframes, I think it would show erroneously as P-frames (the first bug I reported).

(On a separate but connected matter, does anyone know of a free prog that can allow me to see the frame type (I, P, or B) of every frame in a video?  It would be good to be able to confirm when AviDemux begins labelling the frames correctly by an outside source.)

[Jan Gruuthuse]

2013-2-20-23-42-4.sample-SD5-clip-from-Tom-Cook.M2TS:

Code Select Expand

[mpegts @ 0x12569c0] Continuity check failed for pid 0 expected 15 got 0
[mpegts @ 0x12569c0] Continuity check failed for pid 31 expected 12 got 0
[mpegts @ 0x12569c0] Continuity check failed for pid 4113 expected 9 got 0
[mpegts @ 0x12569c0] Continuity check failed for pid 4352 expected 3 got 0
Is this capturing from settop box with haupage capture board? My guess is you need to record the program stream separately from the DVR desktop capturing. Different streams = different info, is not as much a frame issue. If that is not possible you need to use lead in / lead out time for recording program so you can make the cut after switching ...

[TCmullet]

2013-2-20-23-42-4.sample-SD5-clip-from-Tom-Cook.M2TS:

Code Select Expand

[mpegts @ 0x12569c0] Continuity check failed for pid 0 expected 15 got 0
[mpegts @ 0x12569c0] Continuity check failed for pid 31 expected 12 got 0
[mpegts @ 0x12569c0] Continuity check failed for pid 4113 expected 9 got 0
[mpegts @ 0x12569c0] Continuity check failed for pid 4352 expected 3 got 0

How did you get this info?  What tool?  Tell please?

Is this capturing from settop box with haupage capture board? My guess is you need to record the program stream separately from the DVR desktop capturing.

Can't do.  The DVR has the tuner.

Different streams = different info, is not as much a frame issue. If that is not possible you need to use lead in / lead out time for recording program so you can make the cut after switching ...

Sounds like you're tell me to do what I had said I hoped you wouldn't, namely, "don't record that".  I really think a program like this needs to be smart enough to skip over garbage in possibly-damaged files, so that things can be reclaimed.

And I'll still wait for someone to look into the P-frame/I-frame identity issue (the first issue I brought up).  The frame type needs to report accurately.

[Jan Gruuthuse]

Never said don't record that. Problem is switching to and from dvr desktop and channel watched on dvr. DVR desktop is stream output on itself and completly different to the stream from the program. (if I all understand this correctly)
1st solution don't start recording on DVR desktop: start recording when DVR switched to full channel output.
other solution and depending where you get your signal from. For the record I'm only speaking FTA reception cable/dvb-t/dvb-s. No idea if this is possible with iptv (internet TV).
I make my recordings currently on Linux HD receiver with an image on it (+- 165 euro). The streams itself are recorded and not the desktop/video output.

The output was generated trying to rescue the mpeg-ts stream, but failed in this case (was even worse). some times this trick does works:
rename file you wan't to save to input.M2TS.

Code Select Expand

ffmpeg -i input.M2TS -acodec copy -vcodec copy rescue.tsif that fails:

Code Select Expand

ffmpeg -i input.M2TS -acodec copy -vcodec copy rescue.mkv

[TCmullet]

Never said don't record that. Problem is switching to and from dvr desktop and channel watched on dvr. DVR desktop is stream output on itself and completely different to the stream from the program. (if I all understand this correctly)
1st solution don't start recording on DVR desktop: start recording when DVR switched to full channel output.

Well, you ARE saying "don't record that".  And I'll concede (even though the capture program can capture all, and WMP can play back okay).  I'll start capturing AFTER the DVR-menu has fully switched to playing the real show (skipping the first seconds of the real show).  But even there I'm having problems.

Let me back up and report that before starting this thread, I was in contact with Hauppauge customer service.  Upon my allegation of lack of I-frames scattered throughout an SD captured file, he asked me to send him a clip and he'd pass it on to the developer.  I capped a short "live" segment.  I got back a report from them that there WERE I-frames in it.  He showed an image from the *expensive* ($1295) StreamEye software.  I'm attaching the image here.  It shows I-frames present.   I loaded the file into AviD and THAT'S when I discovered that AviD was showing some I-frames with the "P" label.  It's hard to go forward in getting the capture problem solved until the program I'm using to detect I-frames (and subsequent processing thereby--I-frame editing) can correctly report the frame-type.

[TCmullet]

Am having real trouble creating a SHORT clip that shows the problem.  Yet, every time I record a 4-hour DVR-prerecorded show (leaving it overnight with a time-limit set on the capture so it stops shortly after the show is over), it appears there's no I-frames (beyond the first) and AviD either crashes or a seek to the 2nd keyframe seeks to the end of the file, implying there are no I-frames scattered throughout.

Is there a maximum duration over which AviD cannot correctly process?  I have numerous shows over 4-hours that I must capture.  And I must have I-frames throughout, not just the 1st frame.

[Jan Gruuthuse]

Could you try recording in AVCHD .TS format instead of .M2TS? And see if the problem persists? Hauppauge Colossus Recording formats: AVCHD (.TS and .M2TS) and .MP4.
The problem starts here with last clip when you restart a clip on the DVR while you're are recording. This causes some kind of issue for avidemux (signal break, ....)
Still I was able to pass this point, but this is not workable:
Start of clip: press keyboard arrow up and again: you can't go any further at this point Now press left arrow key, now you can move forward and backward only with left/right keyboard arrows. Using mark A and mark B you can now mark section you want to copy/save.
Hope you have better success with .TS format?
This is all from user perspective! I'm just a avidemux user, that is all.
from 2013-2-20-23-42-4.sample-SD5-clip-from-Tom-Cook
part1: http://rapidshare.com/files/124895466/md5part1test.ts
part2: http://rapidshare.com/files/1356645340/md5part2test.ts

[TCmullet]

"Restart a clip" on the DVR fits in to the catagorey of "recording the DVR menu and some content in the same captured clip".  Not simply "restart", but ANY case of capping both DVR menu and show content.  So I'm heeding your advice and NOT starting a capture til after the recorded program is playing.

Right now I'm doing a 2-hr cap following the above new rule, to see if AviD will like it.  I'll have to try out your .TS results later when not capping.

But as far as .mp4 vs .ts vs .m2ts, I have wrestled with that TWICE.  At first, I followed the advice of one Hauppauge guy who suggested .mp4. I followed that advice for days of capping experimentation.  Then learning from here and other places that .mp4 isn't the best for editing, I looked into .ts and .m2ts.  Either should be good according to sources which I can't quote.  I elected .m2ts as the only difference is extra timecode info, which I intuitively suspect could be helpful to some players.  (I figure it NEVER hurts to have timecode info.)  So that's why my 2nd evaluation led me to settle on .m2ts for all files.  I don't understand why .ts would have any advantage to anything.

Have you looked at my SD3 clip??  I allege that AviD is falsely reporting I-frames to be P-frames.  Until that is corrected, we are operating blindly.  We need to get that corrected before plodding onward.

IS THERE a time limit on file size with AviD?  1 hr?  2 hr?   A half-hour SD capture (skipping the DVR-menu of course) worked.  I'm doing a 2-hr cap now.

[Jan Gruuthuse]

Mpeg-ts is what is used in broadcasting: dvb-t/dvb-s/s2. M2TS is more geared to blue-ray and more strict (my impression).
4 hour should not be a problem. Don't think I ever personally worked with over 4 hour recordings myself. Think I had once a multiple part 34 GB recording.

There are three formats which you can choose when recording a video:
.TS which is a generic 'transport stream' compatible with many digital media players
.M2TS which is compatible with the Sony Playstation3
.MP4 which is compatible with the XBox360

[Jan Gruuthuse]

Have you looked at my SD3 clip??  I allege that AviD is falsely reporting I-frames to be P-frames.  Until that is corrected, we are operating blindly.  We need to get that corrected before plodding onward.

Avidemux 2.6.1 (avchd and h264) are not using frames as such. Is a time based editor and not a frame based editor.

[TCmullet]

That may be partially true, Jan.  (I realize that a change from frame-based to time-based occurred between 2.5 and 2.6.)  But it's still true that "frames" show up in the display window, and the "frame type" shows up to the right of the timecode window.  It show I, P, or B depending on whether the *frame* is an I, P, or B.  It's important that it show the correct letter, and I am alleging (with proof provided in my SD3 clip) that it is not.

When you say it is a "time based editor" not a "frame based editor", I say it is a GOP based editor.  It copies GOPs, which I trust you know is a "group of pictures" that starts with an I-frame.  Knowing that I have to edit only at I-frame points (because that's all AviD will allow) makes me frustrated that these SD captures "appear" to not have I-frames scattered throughout.  But then Hauppauge tells me that they ARE there and that I'm blind.  I say, yes, I am blind b/c AviDemux is incorrectly labelling the I-frames as P-frames.  This REALLY REALLY REALLY needs to be corrected as the starting point for further problem solving.  I beg the developer to please correct the mislabelled I-frames.

[TCmullet]

Jan, when you asked if I could try recording in .ts vs .m2ts, I failed to see that you said "AVCHD".  I'm not recording HD, but SD.  The HDs have worked fine.  An I-frame every half second, and the rest are P-frames.  It's SD recordings that either don't have more than 1 I-frame or AviDemux is mislabelling the numerous I-frames as P-frames.  I plead that the mislabellings be corrected, no matter what other issues are being discussed.

[Jan Gruuthuse]

sd3 handles well in 2.6.1 r8483 keyframe every 0.500 seconds. Cuts at start or where need to join: can only be made on the ones selected with up/down keyboard arrow or /
See also for partly explanation: http://www.avidemux.org/smf/index.php/topic,10608.msg57272.html#msg57272.

[TCmullet]

Jan, I looked there and found this:  "Pseudo IDR=I with recovery=0.  As far as navigating is concerned, avidemux uses pseudo IDR as IDR, hence you can use the scrollbar.  To display the frame type, avidemux uses the decoder output and the decoder says it is an I (not IDR) which is considered as a P as far as avidemux is concerned".

Apparently I have to learn what "IDR" and "pseudo IDR" are.  (Sheesh.)  Please help.  So what does he mean by "decoder says it's an I which is considered as a P"??

Can you or someone recommend a tool to determine what kind of frames each of the frames is?  (If AviDemux won't be repaired to show the true state, like I had greatly hoped.)