You dont need to compile avidemux yourself to write your own plugins.
What you need is :

• Avidemux installed on your system
• SDK component activated
• CMake > 3.x and mingw64-32 or 64 installed CMake
• Msys2 installed MSYS
• Mingw64 installed MinGW64
• A demo/sample video filter, available here videofilterdemo_2.7.zip

Make sure you have selected the SDK component during install

It is a good idea to use a short path, i.e. c:\MingW64 and c:\msys64 Make sure you have them both in your path (i.e. c:\Mingw64\mingw64\bin and c:\msys64\usr\bin)

You will need to install cmake version 3.x
Make sure the cmake “bin” folder is in your path

It is best to start here to make sure you have a working environment

Start a MSYS2 shell, go where you unzipped the demo code (i.e. c:\tmp\videoFilterDemo for example )
Create a build folder

 mkdir build && cd build

then generate the makefile

 cmake -G "MSYS Makefiles" -DCMAKE_MAKE_PROGRAM=mingw32-make -DCMAKE_INSTALL_PREFIX="c:\Program Files\Avidemux 2.6 - 64 bits" ..
 

The CMAKE_INSTALL_PREFIX should point to the place where avidemux is

Look for error, it should go without problem.

 make

Upon success, copy the dll in c:\Program Files\Avidemux 2.6 - 64 bits\plugins\videoFilters\
Start avidemux and you should see a logo_sample filter in the menu list

Important/Common problem

If the C++ library used by mingw is more recent than the one used by avidemux itself you'll run into two problem :

• If you have error undefined symbol _imp_xxxxstdcxx11xxxxbasic_stringxxx when compiling , add one line at the very top of CMakeLists.txt

ADD_DEFINITIONS(-D_GLIBCXX_USE_CXX11_ABI=0)

• If you have error at startup “Cannot locate xyz in libsdtc+++”, Overwrite the libstdc++-6.dll in the avidemux folder with the one from mingw, e.g. c:\Mingw64\mingw64\bin\libstdc++-6.dll

Do the fixes above, erase the build folder (build/*) and do it all over again (it takes 5 sec)

You'll need the same requirement as above (i.e. cmake, msys,…)
You *MUST* add msys make, mingw make will not work with netbeans
Open a msys2 shell, and enter

 pacman -S make  
 

There are two parts :

• Configure the toolchain
• Configure the project

The toolchain should be configured as follow :

The important part is to select the make from msys64/usr/bin

Everything is coming from msys, the screenshot is a bit old

Last step is to setup the project :

The important part are :

• At step2 select custom
• At step3, add
  

-G “MSYS Makefiles” -DCMAKE_INSTALL_PREFIX=“c:/Program Files/Avidemux 2.6 - 64 bits”“

Note that the path separator are ”/“ and not “\”!

You can build the sample now.

In case you made a mistake, dont forget to delete the CMakeCache.txt file before retrying