Hi
build 'avidemux_core' with multithreading (for example -j<more than 1>) is broken, make unrecognized files (bad linking?)
http://sl1pkn07.wtf/paste/view/f70f790c
tested with my double xeon x5650 (24threads)
with:
git clone https://github.com/mean00/avidemux2.git avidemux
mkdir -p buildcore
cd build_core
cmake ../avidemux/avidemux_core \
-DFAKEROOT=fakeroot \
-DCMAKE_INSTALL_PREFIX=/usr \
-DCMAKE_BUILD_TYPE=Release
make -j24 VERBOSE=""
make VERBOSE="" DESTDIR="fakeroot" install
only works if set -j to 1 (one thread)
EDIT: also 'make install' with multithreading fails
greetings
anly notice of this?
greetings
yes, that's why the boostrap script force -J 1
It's a broken dependancy somewhere
yep, in bootstrap set to -j1, but I use cmake comands directly instead a use boostrap (for make better split components for distro package maintainer)
all parts works ok (Qt5/Qt4/cli GUI and Qt4/Qt5/cli plugins, core plugins inclusive) with multithread mode except the avidemux_core part.
i think the failure is in ffmpeg build part (ffmpeg with autotools works ok, but not with avidemux cmake wrapper), but need more test
any fix for this?,
23 of my 24 threads machine they had become vague
greetings
It should have been fixed
Scanning dependencies of target libpostproc
make[3]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule.
[ 0%] Built target libpostproc
diff --git a/cmake/admFFmpegBuild.cmake b/cmake/admFFmpegBuild.cmake
index 2550360..81d346c 100644
--- a/cmake/admFFmpegBuild.cmake
+++ b/cmake/admFFmpegBuild.cmake
@@ -300,13 +300,10 @@ endif (FFMPEG_PERFORM_BUILD)
# Build FFmpeg
getFfmpegLibNames("${FFMPEG_SOURCE_DIR}")
-set(ffmpeg_gnumake_executable ${GNUMAKE_EXECUTABLE})
-convertPathToUnix(ffmpeg_gnumake_executable ${BASH_EXECUTABLE})
-configure_file("${AVIDEMUX_TOP_SOURCE_DIR}/cmake/ffmpeg_make.sh.cmake" "${FFMPEG_BINARY_DIR}/ffmpeg_make.sh")
registerFFmpeg("${FFMPEG_SOURCE_DIR}" "${FFMPEG_BINARY_DIR}" 0)
add_custom_target( libavutil_dummy
- COMMAND ${BASH_EXECUTABLE} ffmpeg_make.sh
+ COMMAND $(MAKE)
WORKING_DIRECTORY "${FFMPEG_BINARY_DIR}")
MACRO(FF_ADD_SUBLIB lib)
add_custom_command(
allows parallel build for the bundled FFmpeg. I have not a slightest idea of far-reaching implications of skipping ffmpeg_make.sh.cmake though.
win32 support i think, not sure, it was done by gruntster
perfect!
tnx bro!
Quote from: mean on August 22, 2016, 07:09:50 PM
win32 support i think, not sure, it was done by gruntster
Could you please test if the suggested patch really breaks build on Windows? Because if it doesn't, it would be really nice to have it checked in as-is, maybe with a future follow-up for bootStrap.bash to take the actual number of CPU cores into account instead of hardcoded "-j 2". ffmpeg_make.sh.cmake would become redundant in this case.
TIA
Quote from: eumagga0x2a on August 23, 2016, 12:06:36 PM
maybe with a future follow-up for bootStrap.bash to take the actual number of CPU cores into account instead of hardcoded "-j 2".
e.g.
diff --git a/bootStrap.bash b/bootStrap.bash
index d4ea659..ce67fa1 100644
--- a/bootStrap.bash
+++ b/bootStrap.bash
@@ -16,7 +16,8 @@ debug=0
qt_ext=Qt5
QT_FLAVOR="-DENABLE_QT5=True"
export QT_SELECT=5 # default for ubuntu, harmless for others
-export O_PARAL="-j 2"
+numcpucores=$(grep -c ^core /proc/cpuinfo)
+export O_PARAL="-j $numcpucores"
fail()
{
echo "** Failed at $1**"
use 'nproc' instead for better num of cores detection
Does that distinguish between real core and HT (intel). Quad core + 4 HT comes sometimes up as 8 core?
ââ€Å'ââ€â,¬Ã¢â€Â¤[$]|[sl1pkn07]|[sL1pKn07]|[~]|
ââ€â€Ã¢â€â,¬Ã¢â€â,¬Ã¢â€â,¬Ã¢â€¢Â¼ grep -c ^core /proc/cpuinfo
24
ââ€Å'ââ€â,¬Ã¢â€Â¤[$]|[sl1pkn07]|[sL1pKn07]|[~]|
ââ€â€Ã¢â€â,¬Ã¢â€â,¬Ã¢â€â,¬Ã¢â€¢Â¼ nproc
24
try yourself
my machine is a dual Xeon 6 cores with HT on
jan@jan-Z77-Extreme3:~$ grep -c ^core /proc/cpuinfo
8
jan@jan-Z77-Extreme3:~$ nproc
8
both include the virtual cores (HT)
Quote from: sl1pkn07 on August 23, 2016, 01:58:22 PM
use 'nproc' instead for better num of cores detection
Thanks! Filling in gaps in knowledge is the best part of being a Linux user :)
This will do it:
diff --git a/bootStrap.bash b/bootStrap.bash
index d4ea659..ce4e8d9 100644
--- a/bootStrap.bash
+++ b/bootStrap.bash
@@ -16,7 +16,7 @@ debug=0
qt_ext=Qt5
QT_FLAVOR="-DENABLE_QT5=True"
export QT_SELECT=5 # default for ubuntu, harmless for others
-export O_PARAL="-j 2"
+export O_PARAL="-j $(nproc)"
fail()
{
echo "** Failed at $1**"
First part committed, but done differently
gmake[3]: warning: -jN forced in submake: disabling jobserver mode.
Why hardcoding "-j 4"? I have only two and sl1pkn07 has 24.
Portable way to do it ?
Is $(MAKE) as in
diff --git a/cmake/admFFmpegBuild.cmake b/cmake/admFFmpegBuild.cmake
index 317c7dc..43a3038 100644
--- a/cmake/admFFmpegBuild.cmake
+++ b/cmake/admFFmpegBuild.cmake
@@ -307,7 +307,7 @@ registerFFmpeg("${FFMPEG_SOURCE_DIR}" "${FFMPEG_BINARY_DIR}" 0)
if(CMAKE_HOST_UNIX)
add_custom_target( libavutil_dummy
- COMMAND ${CMAKE_BUILD_TOOL} -j 4 # We assume make or gnumake when host is unix
+ COMMAND $(MAKE) # We assume make or gnumake when host is unix
WORKING_DIRECTORY "${FFMPEG_BINARY_DIR}")
else(CMAKE_HOST_UNIX)
add_custom_target( libavutil_dummy
not portable? Would it fail on Mac?
1- ${MAKE} is not a "true" cmake variable. The true one is CMAKE_BUILD_TOOL or its newer name. Its value can be ninja, make, devenv,.. depending on the platform
2- -j 4 only works if you use unix style make -gnumake)
Quote from: mean on August 24, 2016, 07:58:47 AM
1- ${MAKE} is not a "true" cmake variable.
Actually, FFmpeg compilation with ${MAKE} fails. $(MAKE) works.
QuoteThe true one is CMAKE_BUILD_TOOL or its newer name.
Yes, CMAKE_MAKE_PROGRAM, as CMAKE_BUILD_TOOL is deprecated.
QuoteIts value can be ninja, make, devenv,.. depending on the platform
2- -j 4 only works if you use unix style make -gnumake)
The issue is that this 1) knocks out the make jobserver and 2) hardcodes a concurrency level which may or may not suit the build machine. This is why I would like to find out where the working for me $(MAKE) really fails, because the benefits of using $(MAKE) are obvious while the disadvantages remain sort of abstract.
The following patch doesn't help with the jobserver (seems to be an old cmake bug) but at least sets -jN to the value matching the number of CPU cores:
diff --git a/cmake/admFFmpegBuild.cmake b/cmake/admFFmpegBuild.cmake
index 317c7dc..32cf4b8 100644
--- a/cmake/admFFmpegBuild.cmake
+++ b/cmake/admFFmpegBuild.cmake
@@ -305,9 +305,15 @@ convertPathToUnix(ffmpeg_gnumake_executable ${BASH_EXECUTABLE})
configure_file("${AVIDEMUX_TOP_SOURCE_DIR}/cmake/ffmpeg_make.sh.cmake" "${FFMPEG_BINARY_DIR}/ffmpeg_make.sh")
registerFFmpeg("${FFMPEG_SOURCE_DIR}" "${FFMPEG_BINARY_DIR}" 0)
+include(ProcessorCount)
+ProcessorCount(N)
+if(NOT N EQUAL 0)
+ set(JOBS_IN_PARALLEL -j${N})
+endif(NOT N EQUAL 0)
+
if(CMAKE_HOST_UNIX)
add_custom_target( libavutil_dummy
- COMMAND ${CMAKE_BUILD_TOOL} -j 4 # We assume make or gnumake when host is unix
+ COMMAND ${CMAKE_MAKE_PROGRAM} ${JOBS_IN_PARALLEL} # We assume make or gnumake when host is unix
WORKING_DIRECTORY "${FFMPEG_BINARY_DIR}")
else(CMAKE_HOST_UNIX)
add_custom_target( libavutil_dummy
It is still a hack in comparison to $(MAKE) which propagates concurrency level automagically.