Avidemux Forum

Hi

build 'avidemux_core' with multithreading (for example -j<more than 1>) is broken, make unrecognized files (bad linking?)

http://sl1pkn07.wtf/paste/view/f70f790c

tested with my double xeon x5650 (24threads)

with:
   
    git clone https://github.com/mean00/avidemux2.git avidemux
    mkdir -p buildcore
    cd build_core
    cmake ../avidemux/avidemux_core \
      -DFAKEROOT=fakeroot \
      -DCMAKE_INSTALL_PREFIX=/usr \
      -DCMAKE_BUILD_TYPE=Release
    make -j24 VERBOSE=""
    make VERBOSE="" DESTDIR="fakeroot" install


only works if set -j to 1 (one thread)

EDIT: also 'make install' with multithreading fails

greetings

anly notice of this?

greetings

yes, that's why the boostrap script force -J 1
It's  a broken dependancy  somewhere

yep, in bootstrap set to -j1, but I use cmake comands directly instead a use boostrap (for make better split components for distro package maintainer)

all parts works ok (Qt5/Qt4/cli GUI and Qt4/Qt5/cli plugins, core plugins inclusive) with multithread mode except the avidemux_core part.

i think the failure is in ffmpeg build part (ffmpeg with autotools works ok, but not with avidemux cmake wrapper), but need more test

any fix for this?,

23 of my 24 threads machine they had become vague

greetings

It should have been fixed


Scanning dependencies of target libpostproc
make[3]: warning: jobserver unavailable: using -j1.  Add '+' to parent make rule.
[  0%] Built target libpostproc


diff --git a/cmake/admFFmpegBuild.cmake b/cmake/admFFmpegBuild.cmake
index 2550360..81d346c 100644
--- a/cmake/admFFmpegBuild.cmake
+++ b/cmake/admFFmpegBuild.cmake
@@ -300,13 +300,10 @@ endif (FFMPEG_PERFORM_BUILD)
# Build FFmpeg
getFfmpegLibNames("${FFMPEG_SOURCE_DIR}")

-set(ffmpeg_gnumake_executable ${GNUMAKE_EXECUTABLE})
-convertPathToUnix(ffmpeg_gnumake_executable ${BASH_EXECUTABLE})
-configure_file("${AVIDEMUX_TOP_SOURCE_DIR}/cmake/ffmpeg_make.sh.cmake" "${FFMPEG_BINARY_DIR}/ffmpeg_make.sh")
registerFFmpeg("${FFMPEG_SOURCE_DIR}" "${FFMPEG_BINARY_DIR}" 0)

add_custom_target(                 libavutil_dummy
-    COMMAND ${BASH_EXECUTABLE} ffmpeg_make.sh
+    COMMAND $(MAKE)
                                    WORKING_DIRECTORY "${FFMPEG_BINARY_DIR}")
MACRO(FF_ADD_SUBLIB lib)
         add_custom_command(

allows parallel build for the bundled FFmpeg. I have not a slightest idea of far-reaching implications of skipping ffmpeg_make.sh.cmake though.

win32 support i think, not sure, it was done by gruntster

perfect!

tnx bro!

Quote from: mean on August 22, 2016, 07:09:50 PM
win32 support i think, not sure, it was done by gruntster

Could you please test if the suggested patch really breaks build on Windows? Because if it doesn't, it would be really nice to have it checked in as-is, maybe with a future follow-up for bootStrap.bash to take the actual number of CPU cores into account instead of hardcoded "-j 2". ffmpeg_make.sh.cmake would become redundant in this case.

TIA

Quote from: eumagga0x2a on August 23, 2016, 12:06:36 PM
maybe with a future follow-up for bootStrap.bash to take the actual number of CPU cores into account instead of hardcoded "-j 2".

e.g.
diff --git a/bootStrap.bash b/bootStrap.bash
index d4ea659..ce67fa1 100644
--- a/bootStrap.bash
+++ b/bootStrap.bash
@@ -16,7 +16,8 @@ debug=0
qt_ext=Qt5
QT_FLAVOR="-DENABLE_QT5=True"
export QT_SELECT=5 # default for ubuntu, harmless for others
-export O_PARAL="-j 2"
+numcpucores=$(grep -c ^core /proc/cpuinfo)
+export O_PARAL="-j $numcpucores"
fail()
{
         echo "** Failed at $1**"

use 'nproc' instead for better num of cores detection

Does that distinguish between real core and HT (intel). Quad core + 4 HT comes sometimes up as 8 core?


â”Å'â”â,¬Ã¢â€Â¤[$]|[sl1pkn07]|[sL1pKn07]|[~]|
â””â”â,¬Ã¢â€â,¬Ã¢â€â,¬Ã¢â€¢Â¼  grep -c ^core /proc/cpuinfo
24
â”Å'â”â,¬Ã¢â€Â¤[$]|[sl1pkn07]|[sL1pKn07]|[~]|
â””â”â,¬Ã¢â€â,¬Ã¢â€â,¬Ã¢â€¢Â¼  nproc
24


try yourself

my machine is a dual Xeon 6 cores with HT on

jan@jan-Z77-Extreme3:~$ grep -c ^core /proc/cpuinfo
8
jan@jan-Z77-Extreme3:~$ nproc
8

both include the virtual cores (HT)

Quote from: sl1pkn07 on August 23, 2016, 01:58:22 PM
use 'nproc' instead for better num of cores detection

Thanks! Filling in gaps in knowledge is the best part of being a Linux user  :)

This will do it:

diff --git a/bootStrap.bash b/bootStrap.bash
index d4ea659..ce4e8d9 100644
--- a/bootStrap.bash
+++ b/bootStrap.bash
@@ -16,7 +16,7 @@ debug=0
qt_ext=Qt5
QT_FLAVOR="-DENABLE_QT5=True"
export QT_SELECT=5 # default for ubuntu, harmless for others
-export O_PARAL="-j 2"
+export O_PARAL="-j $(nproc)"
fail()
{
         echo "** Failed at $1**"

First part committed, but done differently

gmake[3]: warning: -jN forced in submake: disabling jobserver mode.

Why hardcoding "-j 4"? I have only two and sl1pkn07 has 24.

Portable way to do it ?

Is $(MAKE) as in

diff --git a/cmake/admFFmpegBuild.cmake b/cmake/admFFmpegBuild.cmake
index 317c7dc..43a3038 100644
--- a/cmake/admFFmpegBuild.cmake
+++ b/cmake/admFFmpegBuild.cmake
@@ -307,7 +307,7 @@ registerFFmpeg("${FFMPEG_SOURCE_DIR}" "${FFMPEG_BINARY_DIR}" 0)

if(CMAKE_HOST_UNIX)
         add_custom_target(         libavutil_dummy
-                                   COMMAND ${CMAKE_BUILD_TOOL}  -j 4 # We assume make or gnumake when host is unix
+                                   COMMAND $(MAKE) # We assume make or gnumake when host is unix
                                    WORKING_DIRECTORY "${FFMPEG_BINARY_DIR}")
else(CMAKE_HOST_UNIX)
         add_custom_target(         libavutil_dummy

not portable? Would it fail on Mac?

1- ${MAKE} is not a "true" cmake variable. The true one is CMAKE_BUILD_TOOL or its newer name. Its value  can be ninja, make, devenv,.. depending on the platform
2- -j 4 only works if you use unix style make -gnumake)

Quote from: mean on August 24, 2016, 07:58:47 AM
1- ${MAKE} is not a "true" cmake variable.

Actually, FFmpeg compilation with ${MAKE} fails. $(MAKE) works.

QuoteThe true one is CMAKE_BUILD_TOOL or its newer name.

Yes, CMAKE_MAKE_PROGRAM, as CMAKE_BUILD_TOOL is deprecated.

QuoteIts value  can be ninja, make, devenv,.. depending on the platform
2- -j 4 only works if you use unix style make -gnumake)

The issue is that this 1) knocks out the make jobserver and 2) hardcodes a concurrency level which may or may not suit the build machine. This is why I would like to find out where the working for me $(MAKE) really fails, because the benefits of using $(MAKE) are obvious while the disadvantages remain sort of abstract.

The following patch doesn't help with the jobserver (seems to be an old cmake bug) but at least sets -jN to the value matching the number of CPU cores:

diff --git a/cmake/admFFmpegBuild.cmake b/cmake/admFFmpegBuild.cmake
index 317c7dc..32cf4b8 100644
--- a/cmake/admFFmpegBuild.cmake
+++ b/cmake/admFFmpegBuild.cmake
@@ -305,9 +305,15 @@ convertPathToUnix(ffmpeg_gnumake_executable ${BASH_EXECUTABLE})
configure_file("${AVIDEMUX_TOP_SOURCE_DIR}/cmake/ffmpeg_make.sh.cmake" "${FFMPEG_BINARY_DIR}/ffmpeg_make.sh")
registerFFmpeg("${FFMPEG_SOURCE_DIR}" "${FFMPEG_BINARY_DIR}" 0)

+include(ProcessorCount)
+ProcessorCount(N)
+if(NOT N EQUAL 0)
+  set(JOBS_IN_PARALLEL -j${N})
+endif(NOT N EQUAL 0)
+
if(CMAKE_HOST_UNIX)
         add_custom_target(         libavutil_dummy
-                                   COMMAND ${CMAKE_BUILD_TOOL}  -j 4 # We assume make or gnumake when host is unix
+                                   COMMAND ${CMAKE_MAKE_PROGRAM} ${JOBS_IN_PARALLEL} # We assume make or gnumake when host is unix
                                    WORKING_DIRECTORY "${FFMPEG_BINARY_DIR}")
else(CMAKE_HOST_UNIX)
         add_custom_target(         libavutil_dummy

It is still a hack in comparison to $(MAKE) which propagates concurrency level automagically.