"Copy/copy" of Mpeg2 program streams absolutely will not work

[TCmullet]

I said this in a slightly related thread.

Another VERY odd quirk about this process of copy/copy of Mpeg2 exists.  Normally my .M2TS files do a copy/copy VERY quickly, hundreds of frames per second.  When this Mpeg-PS stream copy/copy runs, it starts out very slow, maybe around 40 fps, but by the time the "too short" message happens (which is in both 2.6.8 and 2.6.14), it's gone way down to about 8 fps.  WHY?

I hadn't been on the board since then, but use Avidemux greatly.  I didn't upgrade past 2.6.8 til a few weeks ago.  (I had been stuck for a long time in my mindset of having to have extensions of M2TS, when the output muxer is really creating transport streams.  So once I decided to create all captures and resulting edited outputs from Avidemux AS TS FILES, the "extension limitation" of the newer versions would no longer be an obstacle to me, so no reason not to upgrade.  I'm glad I did; 2.7 works so much better!)

I use TS files nearly exclusively.  But lately I've run into big Mpeg2 program streams.  I never had a successful edit of an MPG with Avidemux, but I also have not had the current version.  I just tried it, and it's gotten stuck AGAIN, running very slow "copy/copy", slowing down to several frames per second.  As this is a 2 hour video (27GB), it told me it would take many hours (even days) to complete.

Has ANYONE ever used Avidemux for mpeg2 program streams??  Especially "copy/copy"??   (I really wanted this to work, even if only for GOP-editing.)  It's acting like it's encoding, but there's no encoding in "copy/copy".

[TCmullet]

Just in case anyone wonders, I have this running on a (what I consider) a Super system.  4-core, 8 thread Xeon, w/24GB of ram and a huge Nvidia graphic card taking a strong on-card fan for cooling.  And with adequate disk space.

[dosdan]

So is it size-related? Can you copy a small stream at good speed? If not, provide a file-sharing link so one of us can try it on another system. 

What is showing at the top left for your decoder and for your display driver?


Dan.

[Jan Gruuthuse]

The 27 GB video is on the local hard drive, not on usb media, NAS?
The partition is not a compressed volume? giving more storage then there is really?
The video does play in VLC?

Can you provide media info details of this Mpeg-PS video? Will see if I can create a similar video and test this behaviour.
PS.: AVC 1280x720 stereo AAC, 9 GB (9:49:28.041) duration, takes approx 1h30 conversion to DVD (MPEG-PS is a very strict format)

[eumagga0x2a]

Remuxing a 7.7 GiB big .mpg file with 3.5 hours worth of video in 720x576 resolution took approx. 3 minutes on my low end, 6 years old AMD computer. 27 GB for just two hours implies an insanely high bitrate. As Jan wrote, please provide the MediaInfo analysis of the file, better a sample.

[Jan Gruuthuse]

test performed with: Ubuntu 16.04 64-bit core I7-3770K 32 GB ram
Remark: peak writing at some time 204 MiB/s, visual pausing takes place in terminal output, probably due to caching and transfers i/o to disk.

My test video:

Code Select Expand

General
Complete name                            : ~/Videos/Nordlandsbanen_20120523_Trondheim til Bodø-Spring_nrk.mpg
Format                                   : MPEG-PS
File size                                : 20.1 GiB
Duration                                 : 9h 49mn
Overall bit rate mode                    : Variable
Overall bit rate                         : 4 871 Kbps

Video
ID                                       : 224 (0xE0)
Format                                   : MPEG Video
Format version                           : Version 2
Format profile                           : Main@Main
Format settings, BVOP                    : Yes
Format settings, Matrix                  : Default
Format settings, GOP                     : M=3, N=18
Duration                                 : 9h 49mn
Bit rate mode                            : Variable
Bit rate                                 : 4 550 Kbps
Width                                    : 720 pixels
Height                                   : 480 pixels
Display aspect ratio                     : 4:3
Frame rate                               : 23.976 (24000/1001) fps
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Compression mode                         : Lossy
Bits/(Pixel*Frame)                       : 0.549
Time code of first frame                 : 00:00:00:00
Time code source                         : Group of pictures header
GOP, Open/Closed                         : Open
GOP, Open/Closed of first frame          : Closed
Stream size                              : 18.7 GiB (93%)

Audio
ID                                       : 189 (0xBD)-128 (0x80)
Format                                   : AC-3
Format/Info                              : Audio Coding 3
Mode extension                           : CM (complete main)
Format settings, Endianness              : Big
Muxing mode                              : DVD-Video
Duration                                 : 9h 49mn
Bit rate mode                            : Constant
Bit rate                                 : 224 Kbps
Channel(s)                               : 2 channels
Channel positions                        : Front: L R
Sampling rate                            : 48.0 KHz
Frame rate                               : 31.250 fps (1536 spf)
Compression mode                         : Lossy
Delay relative to video                  : 26h 30mn
Stream size                              : 944 MiB (5%)

on the whole video, on the hour, marked 10 minutes, and cut, saving MPEG-PS with copy/copy (vdpau/vdpau)
saves video in 00:03:16 producing:

Code Select Expand

General
Complete name                            : ~/Videos/Nordlandsbanen_20120523_Trondheim til Bodø-Spring_nrk_edit.mpg
Format                                   : MPEG-PS
File size                                : 17.0 GiB
Duration                                 : 8h 19mn
Overall bit rate mode                    : Variable
Overall bit rate                         : 4 870 Kbps

Video
ID                                       : 224 (0xE0)
Format                                   : MPEG Video
Format version                           : Version 2
Format profile                           : Main@Main
Format settings, BVOP                    : Yes
Format settings, Matrix                  : Default
Format settings, GOP                     : M=3, N=18
Duration                                 : 8h 19mn
Bit rate mode                            : Variable
Bit rate                                 : 4 550 Kbps
Width                                    : 720 pixels
Height                                   : 480 pixels
Display aspect ratio                     : 4:3
Frame rate                               : 23.976 (24000/1001) fps
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Compression mode                         : Lossy
Bits/(Pixel*Frame)                       : 0.549
Time code of first frame                 : 00:00:00:00
Time code source                         : Group of pictures header
GOP, Open/Closed                         : Open
GOP, Open/Closed of first frame          : Closed
Stream size                              : 15.9 GiB (93%)

Audio
ID                                       : 189 (0xBD)-128 (0x80)
Format                                   : AC-3
Format/Info                              : Audio Coding 3
Mode extension                           : CM (complete main)
Format settings, Endianness              : Big
Muxing mode                              : DVD-Video
Duration                                 : 8h 19mn
Bit rate mode                            : Constant
Bit rate                                 : 224 Kbps
Channel(s)                               : 2 channels
Channel positions                        : Front: L R
Sampling rate                            : 48.0 KHz
Frame rate                               : 31.250 fps (1536 spf)
Compression mode                         : Lossy
Delay relative to video                  : 12ms
Stream size                              : 800 MiB (5%)

[Jan Gruuthuse]

same procedure: now lavcodec - RGB, 00:03:12

Code Select Expand

General
Complete name                            : ~/Videos/Nordlandsbanen_20120523_Trondheim til Bodø-Spring_nrk_edit_lavcodec-RGB.mpg
Format                                   : MPEG-PS
File size                                : 17.0 GiB
Duration                                 : 8h 19mn
Overall bit rate mode                    : Variable
Overall bit rate                         : 4 870 Kbps

Video
ID                                       : 224 (0xE0)
Format                                   : MPEG Video
Format version                           : Version 2
Format profile                           : Main@Main
Format settings, BVOP                    : Yes
Format settings, Matrix                  : Default
Format settings, GOP                     : M=3, N=18
Duration                                 : 8h 19mn
Bit rate mode                            : Variable
Bit rate                                 : 4 550 Kbps
Width                                    : 720 pixels
Height                                   : 480 pixels
Display aspect ratio                     : 4:3
Frame rate                               : 23.976 (24000/1001) fps
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Compression mode                         : Lossy
Bits/(Pixel*Frame)                       : 0.549
Time code of first frame                 : 00:00:00:00
Time code source                         : Group of pictures header
GOP, Open/Closed                         : Open
GOP, Open/Closed of first frame          : Closed
Stream size                              : 15.9 GiB (93%)

Audio
ID                                       : 189 (0xBD)-128 (0x80)
Format                                   : AC-3
Format/Info                              : Audio Coding 3
Mode extension                           : CM (complete main)
Format settings, Endianness              : Big
Muxing mode                              : DVD-Video
Duration                                 : 8h 19mn
Bit rate mode                            : Constant
Bit rate                                 : 224 Kbps
Channel(s)                               : 2 channels
Channel positions                        : Front: L R
Sampling rate                            : 48.0 KHz
Frame rate                               : 31.250 fps (1536 spf)
Compression mode                         : Lossy
Delay relative to video                  : 12ms
Stream size                              : 800 MiB (5%)

My conclusion:
- hardware acceleration comes not in play
- editing points could come in to play (Timing differences need to calculate)
- disk i/o and memory transfer comes certainly into play
- and some time I could see time increase in the calculation from 3 minutes, up to 01:48 hour

[TCmullet]

The 27 GB video is on the local hard drive, not on usb media, NAS?
The partition is not a compressed volume? giving more storage then there is really?
The video does play in VLC?

Can you provide media info details of this Mpeg-PS video? Will see if I can create a similar video and test this behaviour.
PS.: AVC 1280x720 stereo AAC, 9 GB (9:49:28.041) duration, takes approx 1h30 conversion to DVD (MPEG-PS is a very strict format)

Gosh, I've "gone thru" so many big Mpeg2 files since I posted this that I don't remember which one it was.  I probably deleted it to make room for more.  But I'm recreating the scenario using a 20GB file I still have.  Yes, a local 7200rpm WD drive, no usb.  I've never had an NAS drive if I understand that an NAS is one of the specialized drives for surveillance systems.  VLC?  Oh, yes, very well.  Your test is a conversion; mine is not but is only "copy/copy".  Was hoping to do GOP editing.

Strict format?  I'm shocked.  Mpeg2 is used by HD TV transmitters at 25mbps or so.  Fast encoding, fast and (relatively simple) decoding (compared to H264), so it made sense when I learned that several years ago.  I thought it was robust enough (if that's the right word) that if transmission was garbled, the stream could continue shortly after the garbling clears up.  I supposed there could be "garblings" when streaming a video to me.  My files are Mpeg2 streamed over internet.  And yes, I bitrated one and it's around 25mbps.  With broadband, I guess content generators consider everyone to have very high speed internet these days.   They are Mpeg2 program streams.  I'll give a mediainfo report.  I use Avidemux mainly for editing TS transport streams (H.264), for which it's been very good usually.  But I've never gotten an Mpeg2 to work.

I've been able to resurrect an oldie but goodie program I've loved for years, MpegVCR.  Doesn't run on newer OS than XP.  And it's been a pain to copy 20-40+ GB files across even my gigabit network.  But it has some error fixing routines.  In most of my files, there are several thousands of "gop errors", which it corrects.  I'm now taking such a corrected file and running thru Avidemux "copy/copy".  I change Muxer from DVD to "free".  (It's 1280x720p30.)   Starts out around 500 fps.  4 min. in it's still clipping along at about 290.  Says 9 min. left.  Hmmm, this is best I've seen.  Maybe my 27GB file had gop-errors and I was hoping Avidemux would clean them up like MpegVCR does.  Btw, VCR cleans up either via the utility OR if you do any edit/save.  But the latter does not reveal that there ever were errors.

At 50% done, it has 7-9 min. left.  This may work.  Could it have been the pesky and ubiquitous gop-errors that "clogged" the brain of Avidemux?

[Jan Gruuthuse]

>8>8 But I'm recreating the scenario using a 20GB file I still have. >8>8
>8>8 Your test is a conversion; mine is not but is only "copy/copy".  >8>8

The 1st file "My test video" was the creation of Mpeg2 PS stream  9h 49mn 20 GB, the source NRK tv, youtube clip.
The following are editing tests:

2nd "on the hour, marked 10 minutes, and cut, saving MPEG-PS with copy/copy (vdpau/vdpau)" hardware accelerated
3rd "same procedure: now lavcodec - RGB, 00:03:12" software

>8>8 Strict format? I'm shocked. Mpeg2 is used by HD TV transmitters at 25mbps or so. Fast encoding, fast and (relatively simple) decoding (compared to H264), so it made sense when I learned that several years ago. >8>8

- You're not allowed as much in MPEG-PS, as you're allowed in other containers.
- Most transmitted HD content now is MPEG-4. MPEG-2 is from 1995's

>8>8 I thought it was robust enough (if that's the right word) that if transmission was garbled, the stream could continue shortly after the garbling clears up. I supposed there could be "garblings" when streaming a video to me.  My files are Mpeg2 streamed over internet. >8>8

Like everyone was lead to believe CD's are robust enough. To find out years later CD's no longer played due to:
- scratches
- Disc rot (phrase describing the tendency of CD or DVD or other optical discs to become unreadable due to physical or chemical deterioration.) source:Disc rot

If you come across MPEG-2 stream that poses problems, try avidemux 2.5.6