How to remove duplicate frames from video that's NOT interlaced?

[Roger Allott]

I have a piece of video that's exactly 30 fps but every fifth frame is a (visually exact) duplicate of its predecessor, suggesting that its true framerate was 24 (or perhaps 23.976) fps. It is not interlaced.

Viewing it in a typical media player looks too jerky for my liking, so I'd like to delete the duplicated frames and end up with my video running at the 'true' speed of 24 fps.

I've used Avidemux for many years and I think it's wonderful for something that's completely free open-source, but getting it to delete duplicate frames has caused me to pull my hair out many times over!

I've tried using Avidemux's deinterlacing filters (Decomb Telecide & Decomb Decimate) even though the source file isn't interlaced. However, my struggles have come from the tutorials out there being a bit light on detail when it comes to selecting the options.

Telecide options:
Strategy - No strategy, 3-2 pulldown, PAL/SECAM, NTSC converted from PAL
Field order - Top, Bottom (this seems completely superfluous as my source isn't interlaced, but I have to select one of these)
Postprocessing - None, None but compute, Postproc on best match, Postproc and show zones, Process image (not fields), Process image (not fields) debug
Try backwards - Never, If still combed, Always
- values for various threshold settings, etc.

Decimate options:
Cycle - integer value - I presume I want 5 in this?
Mode - Discard closer, Replace (interpolate), Discard longer dupe, Pulldown dupe removal
- values for two threshold settings

Based on the above information, could someone guide me on which options I need to select to get what I'm trying to get?

[mean]

true using the "remove ivtc dupe" filter in the nightly

[Roger Allott]

true using the "remove ivtc dupe" filter in the nightly

Thanks. I used the default settings, which gives interesting results, but not what I was wanting.

It removes duplicate frames, but it doubles the amount of time given to the first of them, which results in playback quality that's identical to my source! Instead of constant 30 fps, it gives me variable framerate output of 30 fps and 15 fps.

The closest I've come to getting my desired output came from using only the Decomb Decimate filter (i.e. without Telecide), although for some reason it failed after processing just 8 seconds of footage.

[mean]

Normally no
It creates 24 fps if the source is 30 fps
Could you upload a short sample of your video somewhere ? (mega, rapidshare,...)
10 seconds is enough

[Roger Allott]

Could you upload a short sample of your video somewhere ? (mega, rapidshare,...)
10 seconds is enough

http://depositfiles.com/files/dwwc92r86 (33 MB)

That's a zip file containing:

• source.mp4 - about 20 seconds @ 30 fps with duplicated frames every 5th frame - pattern ABCDD.
• output test decimate cycle 5 only.mp4 @ 24fps without duplicated frames as far as I can see. That was created using Avidemux 2.6.10 (stable build) with only the Decomb Decimate filter applied.
• output test itvc threshold 5.mp4 @ variable framerate with the previously duplicate frames deleted but the duplicated one doubled in duration. This was created using 2.6.12.160508 (the most recent nightly build) with only the remove ivtc dupe filter applied (noise 5, frame rate change Full).

As you can see, both 2 & 3 failed shortly after the source file's first keyframe encountered.

[mean]

Thanks for the sample
It seems the win64 binaries were lagging behing
I've regenerated the nightly, it should work now. Could you confirm ?
I processed the file on linux, and the result looks ok (on linux)

[Roger Allott]

I installed the latest nightly (160517) but unfortunately I'm still getting the same problem, both with frame durations and with it failing shortly after it encounters the first keyframe. I cannot tell whether these two issues are related or completely distinct. I also cannot tell whether the fail is due to the software/codec/filter or a corruption in my source file.

http://depositfiles.com/files/vbekcvy7g (< 5 MB)

[mean]

The problem is actually in the muxer
IF you use mkv or MP4V2, it should work fine (it seems ok here, also checked on win64)

[Roger Allott]

Ah!!! Yes, I can confirm that MP4v2 seems to have done the trick fine, from what I can tell at the moment. Thank you for your help.

A bit of an oddity though is that MediaInfo reports that the output file is still at a constant 30 fps, although if I reimport it into Avidemux and select the Change FPS filter, it's telling me that the current framerate is 26.69. I then run this filter trying to enforce 25 fps, but the output doesn't give me frames that are nice and neatly of consistent 0.040 seconds duration.

The frame times are:
0.302 (I've never understood why the first frame isn't 0.000!)
0.346
0.391
0.435
0.480
0.524
0.569
0.613
.... etc.

see http://depositfiles.com/files/yefxznixc

I can recall a comment from you quite a long time ago that we shouldn't use MP4v2 because it's just there for testing purposes. I presume that's no longer valid, so could you explain what the differences are between it and the main MP4 muxer?

[mean]

you should try using resample fps to 23.976  or 24 fps
it should be smooth

[Roger Allott]

Thanks. I've noticed that you've changed the default for 'Frame Rate Change' in the Remove IVTC Dupe filter from Full to Fast. Could you tell me what each of the options mean?

[mean]

Basically it makes a diff between 2 frames to find which one are closest
In full mode, every line is processed
In fast mode, every 3rd line is processed
In very fast mode, every 9th line is processed

So, fast should be 3 times faster, and "good enough"
Very fast would be even faster, but the accuracy goes down

[Roger Allott]

A quick question relevant to this issue that's probably not worth starting a new thread.

In the IVTC filter, what does the number selected for 'Noise' mean?

[mean]

If you have 2 identical frames, they could be slightly different due to noise
Difference lower than noise (per pixel) are ignored