build 'avidemux_core' with multithreading is broken

Started by sl1pkn07, February 08, 2015, 08:41:07 PM

Previous topic - Next topic

Jan Gruuthuse

jan@jan-Z77-Extreme3:~$ grep -c ^core /proc/cpuinfo
8
jan@jan-Z77-Extreme3:~$ nproc
8

both include the virtual cores (HT)

eumagga0x2a

Quote from: sl1pkn07 on August 23, 2016, 01:58:22 PM
use 'nproc' instead for better num of cores detection

Thanks! Filling in gaps in knowledge is the best part of being a Linux user  :)

This will do it:

diff --git a/bootStrap.bash b/bootStrap.bash
index d4ea659..ce4e8d9 100644
--- a/bootStrap.bash
+++ b/bootStrap.bash
@@ -16,7 +16,7 @@ debug=0
qt_ext=Qt5
QT_FLAVOR="-DENABLE_QT5=True"
export QT_SELECT=5 # default for ubuntu, harmless for others
-export O_PARAL="-j 2"
+export O_PARAL="-j $(nproc)"
fail()
{
         echo "** Failed at $1**"

mean


eumagga0x2a

gmake[3]: warning: -jN forced in submake: disabling jobserver mode.

Why hardcoding "-j 4"? I have only two and sl1pkn07 has 24.

mean


eumagga0x2a

Is $(MAKE) as in

diff --git a/cmake/admFFmpegBuild.cmake b/cmake/admFFmpegBuild.cmake
index 317c7dc..43a3038 100644
--- a/cmake/admFFmpegBuild.cmake
+++ b/cmake/admFFmpegBuild.cmake
@@ -307,7 +307,7 @@ registerFFmpeg("${FFMPEG_SOURCE_DIR}" "${FFMPEG_BINARY_DIR}" 0)

if(CMAKE_HOST_UNIX)
         add_custom_target(         libavutil_dummy
-                                   COMMAND ${CMAKE_BUILD_TOOL}  -j 4 # We assume make or gnumake when host is unix
+                                   COMMAND $(MAKE) # We assume make or gnumake when host is unix
                                    WORKING_DIRECTORY "${FFMPEG_BINARY_DIR}")
else(CMAKE_HOST_UNIX)
         add_custom_target(         libavutil_dummy


not portable? Would it fail on Mac?

mean

1- ${MAKE} is not a "true" cmake variable. The true one is CMAKE_BUILD_TOOL or its newer name. Its value  can be ninja, make, devenv,.. depending on the platform
2- -j 4 only works if you use unix style make -gnumake)


eumagga0x2a

#22
Quote from: mean on August 24, 2016, 07:58:47 AM
1- ${MAKE} is not a "true" cmake variable.

Actually, FFmpeg compilation with ${MAKE} fails. $(MAKE) works.

QuoteThe true one is CMAKE_BUILD_TOOL or its newer name.

Yes, CMAKE_MAKE_PROGRAM, as CMAKE_BUILD_TOOL is deprecated.

QuoteIts value  can be ninja, make, devenv,.. depending on the platform
2- -j 4 only works if you use unix style make -gnumake)

The issue is that this 1) knocks out the make jobserver and 2) hardcodes a concurrency level which may or may not suit the build machine. This is why I would like to find out where the working for me $(MAKE) really fails, because the benefits of using $(MAKE) are obvious while the disadvantages remain sort of abstract.

eumagga0x2a

The following patch doesn't help with the jobserver (seems to be an old cmake bug) but at least sets -jN to the value matching the number of CPU cores:

diff --git a/cmake/admFFmpegBuild.cmake b/cmake/admFFmpegBuild.cmake
index 317c7dc..32cf4b8 100644
--- a/cmake/admFFmpegBuild.cmake
+++ b/cmake/admFFmpegBuild.cmake
@@ -305,9 +305,15 @@ convertPathToUnix(ffmpeg_gnumake_executable ${BASH_EXECUTABLE})
configure_file("${AVIDEMUX_TOP_SOURCE_DIR}/cmake/ffmpeg_make.sh.cmake" "${FFMPEG_BINARY_DIR}/ffmpeg_make.sh")
registerFFmpeg("${FFMPEG_SOURCE_DIR}" "${FFMPEG_BINARY_DIR}" 0)

+include(ProcessorCount)
+ProcessorCount(N)
+if(NOT N EQUAL 0)
+  set(JOBS_IN_PARALLEL -j${N})
+endif(NOT N EQUAL 0)
+
if(CMAKE_HOST_UNIX)
         add_custom_target(         libavutil_dummy
-                                   COMMAND ${CMAKE_BUILD_TOOL}  -j 4 # We assume make or gnumake when host is unix
+                                   COMMAND ${CMAKE_MAKE_PROGRAM} ${JOBS_IN_PARALLEL} # We assume make or gnumake when host is unix
                                    WORKING_DIRECTORY "${FFMPEG_BINARY_DIR}")
else(CMAKE_HOST_UNIX)
         add_custom_target(         libavutil_dummy


It is still a hack in comparison to $(MAKE) which propagates concurrency level automagically.