What does it take to edit growing .ts files?

[Tony Mah]

Hello,

We been using Avidemux for the last 3 Olympic Games.  Volunteers record the competitions and open the files and clip out the performances of specific athletes and give the files to the coaches to show the athletes. Avidemux is easy to use for volunteers and is all we need until recently.

Things are moving faster and we would like to clip the athletes as the file is being recorded and Dropbox the files immediately. This will allow the coach to download the file and show the athlete right after they are done. This means the file is growing and open on the harddrive as we edit the file.

We record a 2h competition on a tv tuner and get a standard mpeg2.ts file that the video is h264 and there are 2LR audio tracks and 6 Dolby digital tracks. The coaches do not care about the audio.

In researching this technique I found out that it will cost 3K USD/channel to do it. The only way is to capture an HDMI output and convert to MXF with time code embedded from system clock and save to harddrive. With this setup any pro editor Premiere, FCP, etc...can open and edit the file.

Is there any way Avidemux can continually Ingest the file every 10 seconds as it is growing so we can edit on the fly? We don't need MXF and we don't need frame accurate and proper time code edits. We need it with in 18 months before the next Games and we can offer a bounty if that is appropriate.

[eumagga0x2a]

Actually you can save parts of a growing file with a bit of handiwork with the current Avidemux without any modifications to the application. Just don't use Windows.

The growing files should be on a locally mounted Linux filesystem like ext4 to avoid issues with file locking. Once a clip has been saved, close the .ts in Avidemux, delete the generated .ts.idx2 index file, reopen the growing .ts file: Avidemux is ready for operation.

Of course, if source files become very large, indexing might take a while, so a way to index only newly added portions of the stream would be nice.

[mean]

It goes against the design of avidemux, i.e. knowing how to access any frame in the video

Using the segmented approach i.e capturing 5mn blocks and editing them might be a solution though

[Tony Mah]

Is there any other solutions?  It is going to cost me 3K per channel I might as well consider paying to have someone code a solution. To be transparent after 3 Olympics we do have a process that is good but not very fun for the volunteers.

1. Record the athletes with game capture card by pressing record and stop
Pro: easy
Con: you always have to be paying attention. If you get distracted you will make a mistake as it is real-time...people have to take lunch and bathroom breaks...

2. Record a 3 hour master for the event as a safety
a) if a mistake is made we can go to the master and grab the clip after the event is over
b) if the clip is time sensitive we start a second recording of the event and stop the first one and clip with Avidemux and dropbox the file.
c.) if necessary we join the 2 videos back into a single master 

If Avidemux architecturally can't easily be updated to continually update the idx2 and read it back in the app then are there other options? I am okay with a segmented approach if it doesn't require closing and reopening the app and deleting files. I don't need frame accurate, if it is accurate to 1 minute it would be fine if that is any easier.

Thanks,

Tony   

[Jan Gruuthuse]

Why looking at avidemux (software) to solve your issues (hardware)?
How many channels do you need to record?
For each channel you require 2 tuners
- 1 for recording master
- 2 nd for interval recording (tuner can normally record 2 streams)
-- interval recording every 10 minutes with 30 seconds lead-in and lead-out
record master -> 1st nas
record interval -> 2nd nas
copy required finished interval recording to computer and process with avidemux

In case your recording computer can't handle the bandwidth of additional tuner cards: a $340 channel solution could be.
2 x SetTopBox(es) Linux E2 Receiver (cable/terrestrial/satellite?). €150/$170 each, 1 tuner model
1st stb records 1 channel (master)
2nd stb records 10 minute interval

record to decent nas (fast storage with fast network connection):
NÃ,° 1  for masters
NÃ,° 2 for interval

Talk to your technical guys and test if this would work.

[eumagga0x2a]

Is it completely unfeasible to educate the volunteers to handle basic file management operations?

Any invocation of a NAS introduces the very issue with file locking which prevents editing of growing files. An internal sat tuner card sufficiently supported on Linux would be the way to go (provided the recorded channel is not encrypted).

[Tony Mah]

Hi Jan,

You might be on to something. Let me see if I can understand what you are saying and correct me if I am wrong.....

1. Find TV recording software that can record a 2h TV show in 10 min intervals or make 12 files.
2. Open the group of files on Avidemux and clip.
3. If we want to refresh, we have to
a. wait 10 minutes
b. close Avidemux, delete the idx2 file
c. Reopen Avidemux and import the files again and repeat.

If this is what you are proposing please confirm as it might work.

eumagga0x2a,

I am not opposed to using Linux as we only require this type of turn around for very few channels and for certain sports like diving....they finish their turn in 10 seconds LOL. For soccer it is a reg recording. Since our entire system is Windows for ease of use, in your solution we would:

1. Find a USB tuner that would work with Linux
2. load Linux on on a laptop 
3. record 2 hour TV show
4. Open the file on Avidemux and clip, if we want to refresh we would
5. close Avidemux and delete idx2
6. reopen avidemux and import file again.

In both cases, a small coding change would do everything we want. Can I get a "reload button" where the file(s) we are editing would be closed, the idx2 deleted and the same file(s) reopened? It should work for both use cases.

Just FYI the volunteer are amazing! We just have 45 channels and 50-75 event per day for summer games and half as much for winter and if you add clipping individual athletes there is a lot of room for errors. We don't want the experience to feel like slave labour where you are chained to the computer.

Any help would be appreciated.

Thanks,

Tony

[eumagga0x2a]

Since our entire system is Windows for ease of use,

I abstain from commenting this...

in your solution we would:

1. Find a USB tuner that would work with Linux

This is possible but a PCIe solution would be preferable. Are you going to record DVB-S2? Please note that if the stream is encrypted, this won't work for you.

2. load Linux on on a laptop

Possible but much worse than building a regular PC considering the usual awful crappiness of firmware in laptops in comparison to mainboards for desktop use.

3. record 2 hour TV show

No need to wait that long. As soon as something of interest has been recorded, you can proceed with 4.

4. Open the file on Avidemux and clip, if we want to refresh we would
5. close Avidemux and delete idx2

No need to close Avidemux, just close the file (which is continuously growing), delete the corresponding idx2 and reopen the file e.g. from the Recent menu in Avidemux.

Can I get a "reload button" where the file(s) we are editing would be closed, the idx2 deleted and the same file(s) reopened? It should work for both use cases.

While I can't code anything myself, this would be indeed sufficient for this setup.

[mean]

Why dont you do segmented capture ?
Capture 20 mn at a time with 2 cards offset by 10 mn
so you would have only 20 mn files to deal with and all files would be independant
If you have 2 cards, if the action is cut in one, take the other one, it will be offset by 10mn

[Tony Mah]

Hello eumagga0x2a and mean,

I think I have a solution, but here is some context. The sporting world lives in the moment, when you dedicate 4 years to have a chance to win a medal you are all in. When you start climbing that ladder to the top of the 10M all that matters is that you can do those tricks and hit the water with no splash, so nothing in the world matters for those few seconds. As a result most the sporting world at the elite level is Mac. The only reason some might have a PC is that they can't afford a MAC or have special analysis software that only works in a PC. 

I just tried Avidemux with an old MACBook and it will open a growing file and index it to the time it downloaded, so thank-you to eumagga0x2a for suggesting this. I am sure I can borrow 4 old MAC's and find some USB tuners to  record directly to the harddrive.

Using a MAC means I don't have to record 2 files in intervals. they can just use the wizard in the recording software and follow the instruction can record a show.

Any chance of getting that refresh button within 12 months for Pyeongcheng?
Click refresh and
1. Video file is closed
2. idx2 file is deleted or renamed if you don't want to delete
3. same file is reopened and indexed

Bonus feature would be to have a cancel index button too, sometimes a tuner is left recording overnight and it is faster to let it index for a short while and close the app and reopen it as most the the recording is not needed.

I am willing to pay something to get this feature if it is appropriate.

There is also a bug in the windows 64bit version, when a recording is left over night it can grow to 25+ Gigs. For some reason I can't edit the file on the 64bit version. You mark the start and end and when you try to clip it nothing happens. Works fine in the 32bit version, bug was first discovered in Sochi Games and still exists last month.

Ease of use is very important as we are trying to do a good job, make it easy and fun and trying to save costs. It is very expensive to fly someone to an Olympics and pay for their food and housing so we are contemplating remote recording and editing using helpdesk software. I can't think of any other software I would use except Avidemux. 

Thanks so much for helping me brainstorm,

Tony

[Jan Gruuthuse]

You don't need to know linux. STB is standalone receiver End User.
Talk to your local techies  sat cable or terrestrial Digital TV recording

Hi Jan,

You might be on to something. Let me see if I can understand what you are saying and correct me if I am wrong.....

1. Find TV recording software that can record a 2h TV show in 10 min intervals or make 12 files.

Settopbox operates like any other TV receiver and can mostly be used with remote. You have Webif interface access thru standard browser

2. Open the group of files on Avidemux and clip.

no: file transfer like filezilla

3. If we want to refresh, we have to
a. wait 10 minutes
b. close Avidemux, delete the idx2 file
c. Reopen Avidemux and import the files again and repeat.

no: you have 10 minutes recordings, and a separate 3 hour master

If this is what you are proposing please confirm as it might work.

Master 3 hour recording should be no problem. I've done 4-5 hour plus recordings
Tesing needs to be done on setting timers in stb for every 10 minutes (6 an hour) these can be daily repeatable.

It all depends on what you are recording and your locality: read 1st 2 lines in this post.

Stay away from usb-dongle recording.
Nas is for storage and fast ether netwerk transport from receivers and to computer(s) 11 Minute video clips: approx 530 MB

more info: Open source set-top box software: http://openpli.org/ here you see some of the popular brands supported by that firmware.
there others to (depending your locality and familiarity of your Tech Savvy People)

[Jan Gruuthuse]

45 channels: how many do you record simultaneous?
Recording is this done at your local site?
Or in the country where the games are? (different broadcast systems/encryptions)
Be aware HDMI recording could/can have copy protection, making recording impossible.