2.6/2.7 AviMuxer audio padding on skew

[KoolAidMan]

The following patch adds audio padding to the AviMuxer on skew. It is capable of handling compounding skews where the audio is first skewed in the positive direction, and then in the negative direction, and rewinds the audio stream to an appropriate location after the padding is complete.

This patch also adds a working GoToTime function to the CopyAudioStream. Previously, the GoToTime function would be called on the base class which had a null audioaccess.

Unfortunately, this patch currently only works if the startTime is at the beginning of the audio stream. I will work on fixing this later.

Code Select Expand


Index: avidemux/common/ADM_audioFilter/src/audiocopy.cpp
===================================================================
--- avidemux/common/ADM_audioFilter/src/audiocopy.cpp (revision 8349)
+++ avidemux/common/ADM_audioFilter/src/audiocopy.cpp (working copy)
@@ -39,7 +39,7 @@
                        ADM_audioStreamCopy(ADM_audioStream *input,uint64_t startTime, int64_t shift); 
virtual                 WAVHeader                *getInfo(void) {return in->getInfo();};
virtual uint8_t         getPacket(uint8_t *buffer,uint32_t *size, uint32_t sizeMax,uint32_t *nbSample,uint64_t *dts);
-//virtual bool            goToTime(uint64_t nbUs);
+virtual bool            goToTime(uint64_t nbUs);
virtual bool             getExtraData(uint32_t *l, uint8_t **d);
          uint64_t        getDurationInUs(void);
          bool            isCBR();
@@ -55,6 +55,11 @@
     this->shift=shift;
}

+bool  ADM_audioStreamCopy::goToTime(uint64_t nbUs)
+{
+    return this->in->goToTime(nbUs);
+}
+
bool ADM_audioStreamCopy::isCBR()
{
     return in->isCBR();
Index: avidemux_plugins/ADM_muxers/muxerAvi/muxerAvi.h
===================================================================
--- avidemux_plugins/ADM_muxers/muxerAvi/muxerAvi.h (revision 8349)
+++ avidemux_plugins/ADM_muxers/muxerAvi/muxerAvi.h (working copy)
@@ -58,6 +58,8 @@
         uint8_t         *videoBuffer;
         audioClock      **clocks;
         uint64_t        audioDelay;
+        int64_t         *padding;
+        uint64_t        *setTimeDts;
public:
                 muxerAvi();
         virtual ~muxerAvi();
Index: avidemux_plugins/ADM_muxers/muxerAvi/muxerAvi.cpp
===================================================================
--- avidemux_plugins/ADM_muxers/muxerAvi/muxerAvi.cpp (revision 8349)
+++ avidemux_plugins/ADM_muxers/muxerAvi/muxerAvi.cpp (working copy)
@@ -40,6 +40,8 @@
     audioPackets=NULL;
     videoBuffer=NULL;
     clocks=NULL;
+    padding=NULL;
+    setTimeDts=NULL;
};
/**
     \fn     muxerAVI
@@ -56,6 +58,9 @@
         delete [] clocks;
         clocks=NULL;
     }
+   
+    delete [] padding;
+    delete [] setTimeDts;
}

/**
@@ -79,9 +84,16 @@
         vStream=s;
         nbAStreams=nbAudioTrack;
         aStreams=a;
-        clocks=new audioClock*[nbAStreams];
+        clocks=new audioClock*[nbAStreams];       
+        padding=new int64_t[nbAStreams];
+        setTimeDts=new uint64_t[nbAStreams];
+       
         for(int i=0;i<nbAStreams;i++)
+        {
             clocks[i]=new audioClock(a[i]->getInfo()->frequency);
+            padding[i]=0;
+            setTimeDts[i]=0;
+        }
         
         return true;
}
@@ -121,12 +133,36 @@
                             aprintf("[Audio] Packet size %"PRIu32" sample:%"PRIu32" dts:%"PRIu64" target :%"PRIu64"\n",
                                             aPacket->sizeInBytes,aPacket->nbSamples,aPacket->dts,targetDts);
                             if(aPacket->dts!=ADM_NO_PTS)
-                                if( abs(aPacket->dts-clk->getTimeUs())>32000)
+                            {
+                                int64_t delta=aPacket->dts-clk->getTimeUs();
+                               
+                                if(abs(delta)>32000)
                                 {
                                     ADM_warning("[AviMuxer] Audio skew!\n");
                                     clk->setTimeUs(aPacket->dts);
-#warning FIXME add padding
+                                   
+                                    if(delta>0)
+                                        padding[audioIndex]+=delta-32000;                                   
+                                    else
+                                        padding[audioIndex]+=delta+32000;
+                                   
+                                    if(setTimeDts[audioIndex])
+                                        setTimeDts[audioIndex]+=delta;
+                                    else if(padding[audioIndex]>32000)
+                                        setTimeDts[audioIndex]=aPacket->dts;
                                 }
+                            }
+                           
+                            if(padding[audioIndex]>=32000)
+                                padding[audioIndex]-=32000;
+                            else if(setTimeDts[audioIndex])
+                            {
+                                ADM_info("[AviMuxer] Rewinding audio stream\n");
+                                a->goToTime(setTimeDts[audioIndex]);
+                                clk->setTimeUs(setTimeDts[audioIndex]);
+                                setTimeDts[audioIndex]=0;
+                            }
+                           
                             aPacket->present=true;
                     }
                     // We now have a packet stored


[KoolAidMan]

I believe the issue with the patch currently is due to the inability to get the time in us from the audio stream. The dts alone is not enough to be able to know where I need to seek to once I am done padding. I see there is a commented out getTime function on ADM_audioStream-- and it would be helpful if I could use this to get the actual time from the audioStream. I will investigate later.

[KoolAidMan]

update: I'm getting closer to getting this to work. This still does not work quite right. This is what I currently have.

I'm getting some weird numbers when calling _file.getPos() inside of dmxPSPacket.cpp which I have wired to be called downstream when getPos is called on the audioStream. Note that I have some really verbose debug logging in this code.

Code Select Expand


Index: avidemux/common/ADM_audioFilter/src/audiocopy.cpp
===================================================================
--- avidemux/common/ADM_audioFilter/src/audiocopy.cpp (revision 8349)
+++ avidemux/common/ADM_audioFilter/src/audiocopy.cpp (working copy)
@@ -39,7 +39,7 @@
                        ADM_audioStreamCopy(ADM_audioStream *input,uint64_t startTime, int64_t shift); 
virtual                 WAVHeader                *getInfo(void) {return in->getInfo();};
virtual uint8_t         getPacket(uint8_t *buffer,uint32_t *size, uint32_t sizeMax,uint32_t *nbSample,uint64_t *dts);
-//virtual bool            goToTime(uint64_t nbUs);
+virtual bool            goToTime(uint64_t nbUs);
virtual bool             getExtraData(uint32_t *l, uint8_t **d);
          uint64_t        getDurationInUs(void);
          bool            isCBR();
@@ -55,6 +55,11 @@
     this->shift=shift;
}

+bool  ADM_audioStreamCopy::goToTime(uint64_t nbUs)
+{
+    return this->in->goToTime(nbUs);
+}
+
bool ADM_audioStreamCopy::isCBR()
{
     return in->isCBR();
Index: avidemux/common/ADM_editor/include/ADM_edAudioTrackFromVideo.h
===================================================================
--- avidemux/common/ADM_editor/include/ADM_edAudioTrackFromVideo.h (revision 8349)
+++ avidemux/common/ADM_editor/include/ADM_edAudioTrackFromVideo.h (working copy)
@@ -45,6 +45,10 @@
             virtual uint8_t     getPacket(uint8_t *buffer,uint32_t *size, uint32_t sizeMax,uint32_t *nbSample,uint64_t *dts);
             virtual bool         getPCMPacket(float  *dest, uint32_t sizeMax, uint32_t *samples,uint64_t *odts);
             virtual bool         goToTime(uint64_t nbUs);
+            virtual uint8_t      getTime(uint64_t *nbUs)
+                                 {
+                                     return getCurrentTrack()->stream->getTime(nbUs);
+                                 }
                     bool          getExtraData(uint32_t *l, uint8_t **d);
                     uint64_t      getDurationInUs(void);
                     uint8_t    getAudioStream(ADM_audioStream **audio);
Index: avidemux_plugins/ADM_muxers/muxerAvi/muxerAvi.h
===================================================================
--- avidemux_plugins/ADM_muxers/muxerAvi/muxerAvi.h (revision 8349)
+++ avidemux_plugins/ADM_muxers/muxerAvi/muxerAvi.h (working copy)
@@ -58,6 +58,8 @@
         uint8_t         *videoBuffer;
         audioClock      **clocks;
         uint64_t        audioDelay;
+        int64_t         *padding;
+        uint64_t        *setTimeDts;
public:
                 muxerAvi();
         virtual ~muxerAvi();
Index: avidemux_plugins/ADM_muxers/muxerAvi/muxerAvi.cpp
===================================================================
--- avidemux_plugins/ADM_muxers/muxerAvi/muxerAvi.cpp (revision 8349)
+++ avidemux_plugins/ADM_muxers/muxerAvi/muxerAvi.cpp (working copy)
@@ -40,6 +40,8 @@
     audioPackets=NULL;
     videoBuffer=NULL;
     clocks=NULL;
+    padding=NULL;
+    setTimeDts=NULL;
};
/**
     \fn     muxerAVI
@@ -56,6 +58,9 @@
         delete [] clocks;
         clocks=NULL;
     }
+   
+    delete [] padding;
+    delete [] setTimeDts;
}

/**
@@ -79,9 +84,16 @@
         vStream=s;
         nbAStreams=nbAudioTrack;
         aStreams=a;
-        clocks=new audioClock*[nbAStreams];
+        clocks=new audioClock*[nbAStreams];       
+        padding=new int64_t[nbAStreams];
+        setTimeDts=new uint64_t[nbAStreams];
+       
         for(int i=0;i<nbAStreams;i++)
+        {
             clocks[i]=new audioClock(a[i]->getInfo()->frequency);
+            padding[i]=0;
+            setTimeDts[i]=0;
+        }
         
         return true;
}
@@ -121,12 +133,56 @@
                             aprintf("[Audio] Packet size %"PRIu32" sample:%"PRIu32" dts:%"PRIu64" target :%"PRIu64"\n",
                                             aPacket->sizeInBytes,aPacket->nbSamples,aPacket->dts,targetDts);
                             if(aPacket->dts!=ADM_NO_PTS)
-                                if( abs(aPacket->dts-clk->getTimeUs())>32000)
+                            {
+                                int64_t delta=aPacket->dts-clk->getTimeUs();
+                               
+                                if(abs(delta)>32000)
                                 {
                                     ADM_warning("[AviMuxer] Audio skew!\n");
-                                    clk->setTimeUs(aPacket->dts);
-#warning FIXME add padding
+                                    ADM_info("delta: %d\n",delta);
+                                    clk->setTimeUs(aPacket->dts);                             
+
+                                    padding[audioIndex]+=delta-32000;
+                                    ADM_info("padding 1: %d\n",padding[audioIndex]);
+                                   
+                                    if(setTimeDts[audioIndex])
+                                    {
+                                        setTimeDts[audioIndex]+=delta;
+                                        ADM_info("setTimeDts 1: %d\n",setTimeDts[audioIndex]);
+                                    }
+                                    else
+                                    {
+                                        uint64_t time;
+                                       
+                                        if(a->getTime(&time))
+                                        {
+                                            setTimeDts[audioIndex]=time;
+                                            ADM_info("setTimeDts 2: %d\n",setTimeDts[audioIndex]);
+                                        }
+                                    }
                                 }
+                            }
+                           
+                            if(padding[audioIndex]>=32000)
+                            {
+                                padding[audioIndex]-=32000;
+                                ADM_info("padding 2: %d\n",padding[audioIndex]);
+                            }
+                            else if(setTimeDts[audioIndex])
+                            {
+                                ADM_info("[AviMuxer] Rewinding audio stream\n");
+                                a->goToTime(setTimeDts[audioIndex]);
+                                clk->setTimeUs(setTimeDts[audioIndex]);
+                                setTimeDts[audioIndex]=0;
+                                ADM_info("setTimeDts 3: %d\n",setTimeDts[audioIndex]);
+                               
+                                if(padding<0)
+                                {
+                                    padding=0;
+                                    ADM_info("padding 3: %d\n",padding[audioIndex]);
+                                }
+                            }
+                           
                             aPacket->present=true;
                     }
                     // We now have a packet stored
Index: avidemux_plugins/ADM_demuxers/MpegPS/ADM_ps.h
===================================================================
--- avidemux_plugins/ADM_demuxers/MpegPS/ADM_ps.h (revision 8349)
+++ avidemux_plugins/ADM_demuxers/MpegPS/ADM_ps.h (working copy)
@@ -69,6 +69,11 @@
                 virtual uint64_t  getDurationInUs(void) ;
                                     /// Go to a given time
                 virtual bool      goToTime(uint64_t timeUs) ;
+                                    /// Get Position
+                virtual uint64_t  getPos()
+                                      {
+                                          return demuxer.getPos();
+                                      }
                                     /// Get a packet
                 virtual bool      getPacket(uint8_t *buffer, uint32_t *size, uint32_t maxSize,uint64_t *dts);

Index: avidemux_core/ADM_coreAudio/include/ADM_audioStream.h
===================================================================
--- avidemux_core/ADM_coreAudio/include/ADM_audioStream.h (revision 8349)
+++ avidemux_core/ADM_coreAudio/include/ADM_audioStream.h (working copy)
@@ -99,8 +99,13 @@
virtual uint8_t         getPacket(uint8_t *buffer,uint32_t *size, uint32_t sizeMax,uint32_t *nbSample,uint64_t *dts);
/// Go to a given time, in microseconds
virtual bool            goToTime(uint64_t nbUs);
-/// Returns current time in us. Not used.
-//virtual uint8_t         getTime(uint64_t *nbUs);
+/// Returns current time in us.
+virtual uint8_t         getTime(uint64_t *nbUs)
+                        {
+                            if(!access) return 0;
+                            *nbUs=access->getPos();
+                            return 1;
+                        }
/// Returns extra configuration data
virtual bool            getExtraData(uint32_t *l, uint8_t **d);
/// Returns or compute duration. If the access cannot provide it, it will be computed here
Index: avidemux_core/ADM_coreDemuxerMpeg/src/dmxPSPacket.cpp
===================================================================
--- avidemux_core/ADM_coreDemuxerMpeg/src/dmxPSPacket.cpp (revision 8349)
+++ avidemux_core/ADM_coreDemuxerMpeg/src/dmxPSPacket.cpp (working copy)
@@ -82,7 +82,9 @@
*/
uint64_t    psPacket::getPos(void)
{
-    return 0;
+    uint64_t pos;
+    _file->getpos(&pos);
+    return pos;
}
/**
     \fn setPos


one thing to note-- often times when getPos is called, it causes an exception due to an ADM_Assert(0) in the ADM_audioAccess base class. I checked into all of the audioAccess and here is the status of the getPos() function.

• ADM_audioAccess - explodes.
• ADMAudioFilter_Access - returns 0.
• ADM_audioAccessFile - seems to work.
• asfAudioAccess - returns 0.
• ADM_avsAccess - not implemented. Calls base which explodes.
• ADM_flvAccess - not implemented. Calls base which explodes.
• mkvAccess - not implemented. Calls base which explodes.
• ADM_mp4AudioAccess - implemented, but also blows up due to an assert(0)? lol
• ADM_tsAccess - not implemented. Calls base which explodes. Doesn't look terribly difficult to fix-- possibly same as fix above.
• ADM_psAccess - fixed above.
• ADM_aviAudioAccess - seems to work.
• ADM_audioAccess_thread - returns 0.

[KoolAidMan]

All right mean. This is currently what I am seeing.

 [fillAudio]  [AviMuxer] Audio skew!
 [fillAudio]  delta: 414097
[audioClock] Drift detected :0 vs 414097, delta=414097
 [fillAudio]  padding : 382097
 [fillAudio]  setTimeDts : 280633

Ok, so basically, I have audio skew with a delta of 414097. The problem is, when I call getTime() on the audioStream (ADM_audioStreamCopy.getTime()->ADM_edAudioTrackFromVideo.getTime()->ADM_psAccess.getPos()->psPacket.getPos()->fileParser.getPos()) it returns 280633.

Is fileParser.getPos() not guaranteed to be accurate, or is the delta somehow wrong?

update: I looked through the fileParser, and getPos seems to work fine. My guess is the dts is being modified by something while I am getting the packet-- perhaps incorrectly? I will look into this next time I get a chance.

[mean]

getTime() in that case is unreliable
as it assumes completely CBR file
i.e.
time=position/bytePerSec

In PS& TS file in particular this is unreliable and should not be used
That's why getTime() has been deprecated

If you still want it, the proper way is to set getTime() to the DTS of the last packet + packetDuration
(and deal with case where DTS=unknown)