Skip to content

Conversation

@davidrohr
Copy link
Collaborator

Not yet finished, but let's see what the CI says...

@davidrohr davidrohr requested a review from a team as a code owner March 26, 2025 22:16
@github-actions
Copy link
Contributor

REQUEST FOR PRODUCTION RELEASES:
To request your PR to be included in production software, please add the corresponding labels called "async-" to your PR. Add the labels directly (if you have the permissions) or add a comment of the form (note that labels are separated by a ",")

+async-label <label1>, <label2>, !<label3> ...

This will add <label1> and <label2> and removes <label3>.

The following labels are available
async-2023-pbpb-apass4
async-2023-pp-apass4
async-2024-pp-apass1
async-2022-pp-apass7
async-2024-pp-cpass0
async-2024-PbPb-apass1
async-2024-ppRef-apass1
async-2024-PbPb-apass2
async-2023-PbPb-apass5

@alibuild
Copy link
Collaborator

alibuild commented Mar 26, 2025

Error while checking build/O2/fullCI_slc9 for 6455bcb at 2025-03-27 08:24:

## sw/BUILD/O2-RTC-test-latest/log
clang++: error: unknown argument: '-mdaz-ftz'
clang++: error: unknown argument: '-mdaz-ftz'
clang++: error: unknown argument: '-mdaz-ftz'


## sw/BUILD/o2checkcode-latest/log
--
========== List of errors found ==========
++ GRERR=0
++ grep -v clang-diagnostic-error error-log.txt
++ grep ' error:'
grep: error-log.txt: binary file matches
++ GRERR=1
++ [[ 1 == 0 ]]
++ mkdir -p /sw/INSTALLROOT/d87a91743b5a0097228b8350398e554004b509a3/slc9_x86-64/o2checkcode/1.0-local10/etc/modulefiles
++ cat
--

Full log here.

@alibuild
Copy link
Collaborator

Error while checking build/O2/fullCI_slc9 for 2bdcecb at 2025-03-27 15:22:

## sw/BUILD/O2-latest/log
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:65:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:71:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:78:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:84:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:94:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:100:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:107:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:113:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:121:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:127:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:137:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:143:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:150:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:160:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:166:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:175:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:181:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:190:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:196:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:204:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:210:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:218:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:224:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:232:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:238:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:246:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:252:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:260:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:266:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:274:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:280:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:287:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:293:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:300:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:306:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:313:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:319:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:336:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:348:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:354:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:362:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:368:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:376:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:382:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:390:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:396:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:403:1: error: a function-definition is not allowed here before '{' token
/sw/slc9_x86-64/GCC-Toolchain/v14.2.0-alice2-1/lib/gcc/x86_64-unknown-linux-gnu/14.2.0/include/mmintrin.h:409:1: error: a function-definition is not allowed here before '{' token
[0 more errors; see full log]

Full log here.

@davidrohr
Copy link
Collaborator Author

Now ready to be merged once green

@davidrohr
Copy link
Collaborator Author

onnxruntime.log
@ChSonnabend @ktf : And here is the first fallout from ONNXRuntime, it seems to break o2_slc9 AliceO2 CI. Log is here.
I don't fully understand why it fails from the log.

I'd assume it is due to protobuf, see the log part below.

But then I also see something about CUDA:

NVCC_ERROR = 
NVCC_OUT = no such file or directory

despite it having in the CMake command -Donnxruntime_USE_CUDA=OFF. Perhaps we should get rid of the -Donnxruntime_CUDA_HOME=/usr/local/cuda...

CMake Error at /sw/slc9_x86-64/CMake/v3.31.6-1/share/cmake-3.31/Modules/FindProtobuf.cmake:613 (file):
  file STRINGS file
  "/local/workspace/DailyBuilds/DailyO2Physics-slc9/daily-tags.W565Kdf8oJ/slc9_x86-64/protobuf/v29.3-1/include/google/protobuf/stubs/common.h"
  cannot be read.
Call Stack (most recent call first):
  /sw/slc9_x86-64/onnx/v1.17.0-1/lib64/cmake/ONNX/ONNXConfig.cmake:11 (find_package)
  /sw/slc9_x86-64/CMake/v3.31.6-1/share/cmake-3.31/Modules/FetchContent.cmake:2472 (find_package)
  /sw/slc9_x86-64/CMake/v3.31.6-1/share/cmake-3.31/Modules/FetchContent.cmake:2362 (__FetchContent_MakeAvailable_find_package)
  external/helper_functions.cmake:22 (FetchContent_MakeAvailable)
  external/onnxruntime_external_deps.cmake:533 (onnxruntime_fetchcontent_makeavailable)
  CMakeLists.txt:694 (include)


CMake Error at /sw/slc9_x86-64/CMake/v3.31.6-1/share/cmake-3.31/Modules/FindProtobuf.cmake:619 (math):
  math cannot parse the expression: " / 1000000": syntax error, unexpected
  exp_DIVIDE (2).
Call Stack (most recent call first):
  /sw/slc9_x86-64/onnx/v1.17.0-1/lib64/cmake/ONNX/ONNXConfig.cmake:11 (find_package)
  /sw/slc9_x86-64/CMake/v3.31.6-1/share/cmake-3.31/Modules/FetchContent.cmake:2472 (find_package)
  /sw/slc9_x86-64/CMake/v3.31.6-1/share/cmake-3.31/Modules/FetchContent.cmake:2362 (__FetchContent_MakeAvailable_find_package)
  external/helper_functions.cmake:22 (FetchContent_MakeAvailable)
  external/onnxruntime_external_deps.cmake:533 (onnxruntime_fetchcontent_makeavailable)
  CMakeLists.txt:694 (include)


CMake Error at /sw/slc9_x86-64/CMake/v3.31.6-1/share/cmake-3.31/Modules/FindProtobuf.cmake:620 (math):
  math cannot parse the expression: " / 1000 % 1000": syntax error,
  unexpected exp_DIVIDE (2).
Call Stack (most recent call first):
  /sw/slc9_x86-64/onnx/v1.17.0-1/lib64/cmake/ONNX/ONNXConfig.cmake:11 (find_package)
  /sw/slc9_x86-64/CMake/v3.31.6-1/share/cmake-3.31/Modules/FetchContent.cmake:2472 (find_package)
  /sw/slc9_x86-64/CMake/v3.31.6-1/share/cmake-3.31/Modules/FetchContent.cmake:2362 (__FetchContent_MakeAvailable_find_package)
  external/helper_functions.cmake:22 (FetchContent_MakeAvailable)
  external/onnxruntime_external_deps.cmake:533 (onnxruntime_fetchcontent_makeavailable)
  CMakeLists.txt:694 (include)


CMake Error at /sw/slc9_x86-64/CMake/v3.31.6-1/share/cmake-3.31/Modules/FindProtobuf.cmake:621 (math):
  math cannot parse the expression: " % 1000": syntax error, unexpected
  exp_MOD (2).
Call Stack (most recent call first):
  /sw/slc9_x86-64/onnx/v1.17.0-1/lib64/cmake/ONNX/ONNXConfig.cmake:11 (find_package)
  /sw/slc9_x86-64/CMake/v3.31.6-1/share/cmake-3.31/Modules/FetchContent.cmake:2472 (find_package)
  /sw/slc9_x86-64/CMake/v3.31.6-1/share/cmake-3.31/Modules/FetchContent.cmake:2362 (__FetchContent_MakeAvailable_find_package)
  external/helper_functions.cmake:22 (FetchContent_MakeAvailable)
  external/onnxruntime_external_deps.cmake:533 (onnxruntime_fetchcontent_makeavailable)
  CMakeLists.txt:694 (include)


CMake Warning at /sw/slc9_x86-64/CMake/v3.31.6-1/share/cmake-3.31/Modules/FindProtobuf.cmake:646 (message):
  Protobuf compiler version 29.3 doesn't match library version
  ERROR.ERROR.ERROR
Call Stack (most recent call first):
  /sw/slc9_x86-64/onnx/v1.17.0-1/lib64/cmake/ONNX/ONNXConfig.cmake:11 (find_package)
  /sw/slc9_x86-64/CMake/v3.31.6-1/share/cmake-3.31/Modules/FetchContent.cmake:2472 (find_package)
  /sw/slc9_x86-64/CMake/v3.31.6-1/share/cmake-3.31/Modules/FetchContent.cmake:2362 (__FetchContent_MakeAvailable_find_package)
  external/helper_functions.cmake:22 (FetchContent_MakeAvailable)
  external/onnxruntime_external_deps.cmake:533 (onnxruntime_fetchcontent_makeavailable)
  CMakeLists.txt:694 (include)

-- Found Protobuf: /sw/slc9_x86-64/protobuf/v29.3-1/lib/libprotobuf.a (found version "ERROR.ERROR.ERROR")

@ChSonnabend
Copy link
Collaborator

ChSonnabend commented Mar 27, 2025

Just found it too while building on the EPNs and contacted Giulio already

-Donnxruntime_CUDA_HOME=/usr/local/cuda: I can try tomorrow if this builds on my linux machine without. When I built with the docker image last time, I needed this for it to work. Maybe something has changed in the meantime

DDS fails aswell

In file included from /scratch/csonnabe/MyO2/sw/BUILD/f064a5d905e102ae19f1d412a1e47ba1b56f30d5/DDS/proto/submit_info.pb.h:20, from /scratch/csonnabe/MyO2/sw/BUILD/f064a5d905e102ae19f1d412a1e47ba1b56f30d5/DDS/proto/submit_info.pb.cc:6: /scratch/csonnabe/MyO2/sw/slc9_x86-64/protobuf/v29.3-1/include/google/protobuf/io/coded_stream.h:107:10: fatal error: absl/log/vlog_is_on.h: No such file or directory 107 | #include "absl/log/vlog_is_on.h" // Replace with vlog_is_on.h after Abseil LTS 20240722

@davidrohr davidrohr merged commit 90f3763 into AliceO2Group:dev Mar 27, 2025
10 of 11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants