legate.core/src/core/utilities/type_traits.h:85:46: error: ‘complex’ was not declared in this scope

[1193749292]

Legion Build Commands:
cmake .. -S /home/xxx/hpc3/legion/ -D CMAKE_INSTALL_PREFIX=../install -D REALM_USE_OPENMP=ON -D Legion_USE_OpenMP=ON -D REALM_OPENMP_SYSTEM_RUNTIME=ON -D Legion_OpenMP_SYSTEM_RUNTIME=ON -D REALM_USE_MPI=ON -DLegion_NETWORKS=mpi -D Legion_MPI_INTEROP=ON -DLEGION_REDOP_HALF=ON -B ../build

Legate Installation Commands:
./install.py --network mpi --openmp --with-legion /home/xxx/hpc3/legion/build/ -v --nocuda --march native

Description: Ubuntu 20.04.5 LTS | aarch64

The following files are the installation output information.
legate.core-install_output.txt

[manopapad]

Are you on branch control_replication for Legion?

[1193749292]

@manopapad The legion branch is legion-22.12. I also tested the control_replication branch. The difference is that there is no error about legion'OutputRegion'. Everything else is the same.

legate.core-legion-control_replication_install_output.txt

[manopapad]

I believe this is happening because you are missing -DLegion_REDOP_COMPLEX=ON from your Legion build command (and possibly also -DLegion_BUILD_BINDINGS=ON and -DLegion_USE_Python=ON). Can you try adding that and building again?

I submitted a PR to document the required Legion branch and configuration options #670, and will be in touch with the Legion devs to try and export these settings, so we can check for their presence when a user passes in a pre-built installation of Legion.

[1193749292]

@manopapad It's useful. I tried the following methods and the results were the same.
legate.core/src/core/utilities/type_traits.h

However, an error is reported after the legate is successfully installed.

---

Executing: " cmake --install /opt/home/xxx/hpc3/legion/build/bindings/python --prefix /home/xxx/anaconda3 " with {}
CMake Error: Error processing file: /home/xxx/hpc3/legion/build/bindings/python/cmake_install.cmake

An error is reported when the legate is running.

[manopapad]

Have you tried rebuilding Legion with the flags I suggested? I suggest you remove your old Legion installation and rebuild with:

cmake .. -S /home/xxx/hpc3/legion/ -D CMAKE_INSTALL_PREFIX=../install -D REALM_USE_OPENMP=ON -D Legion_USE_OpenMP=ON -D REALM_OPENMP_SYSTEM_RUNTIME=ON -D Legion_OpenMP_SYSTEM_RUNTIME=ON -D REALM_USE_MPI=ON -DLegion_NETWORKS=mpi -D Legion_MPI_INTEROP=ON -DLEGION_REDOP_HALF=ON -DLegion_REDOP_COMPLEX=ON -DLegion_BUILD_BINDINGS=ON -DLegion_USE_Python=ON -B ../build

Then install legate.core using --with-legion.

Or alternatively you can let legate.core build Legion on its own, by not passing a --with-legion or --legion-src-dir.

[1193749292]

@manopapad Rebuild the legion based on your suggestions. After the cmake is successfully executed, the following error message is displayed when the make command is executed:

[manopapad]

It this happening purely during Legion compilation? If so, could you please report this issue on the Legion bug tracker? I suspect it might have something to do with these flags -D REALM_OPENMP_SYSTEM_RUNTIME=ON -D Legion_OpenMP_SYSTEM_RUNTIME=ON.

[1193749292]

@manopapad Thank you for your advice, someone here has replied.
StanfordLegion/legion#1456

[1193749292]

StanfordLegion/legion#1456
https://gitlab.com/StanfordLegion/legion/-/commit/78c5f3ab9c4ea1393183ca135fbca3c403a3113b
This is the solution to legion_python. Modify python_internal.h and python_module.cc, add

-DCMAKE_BUILD_TYPE=Release

and rebuild the legion. Otherwise, the following error message is displayed:

---

legion_python: /root/xxx/legion-control_replication/runtime/realm/timers.inl:72: static long long int Realm::Clock: :current_time_in_nanoseconds(bool): Assertion ‘nanoseconds <= uint64_t(LLONG_MAX)' failed.
Signal legion_python: /root/xxx/legion-control_replication/runtime/realm/timers.inl:72: static long long int Realm: :Clock: :current_time_in_nanoseconds(bool): Assertion ‘nanoseconds <= uint64_t(LLONG_MAX)' failed.
6 received by node 0, process 60468 (thread 7f8112e19880) obtaining backtrace

---

After the legate is successfully installed, run the legate command. The following error information is displayed:

---

[0 - 7f4320297880] -1555852541.985930 {6}{python}: python exception occurred within task:
Traceback (most recent call last):
File "/root/miniconda3/envs/legion-xxx/lib/python3.9/site-packages/legion_top.py", line 409, in legion_python_main
c.legion_task_preamble(
File "/root/miniconda3/envs/legion-xxx/lib/python3.9/site-packages/cffi/api.py", line 912, in _getattr_
make_accessor(name)
File "/root/miniconda3/envs/legion-xxx/lib/python3.9/site-packages/cffi/api.py", line 908, in make_accessor
accessors[name](name)
File "/root/miniconda3/envs/legion-xxx/lib/python3.9/site-packages/cffi/api.py", line 838, in accessor_function
value = backendlib.load_function(BType, name)
AttributeError: function/symbol 'legion_task_preamble' not found in library '/root/miniconda3/envs/legion-xxx/lib/liblegion_canonical_python.so': /root/miniconda3/envs/legion-xxx/lib/liblegion_canonical_python.so: undefined symbol: legion_task_preamble
legion_python: /root/xxx/legion-control_replication/runtime/realm/python/python_module.cc:1006: virtual void Realm::LocalPythonProcessor::execute_task(Realm::Processor::TaskFuncID, const Realm::ByteArrayRef&): Assertion `0' failed.
Signal 6 received by node 0, process 3373 (thread 7f4320297880) - obtaining backtrace
Signal 6 received by process 3373 (thread 7f4320297880) at: stack trace: 14 frames
[0] = /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f4423709980]
[1] = /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7) [0x7f4422838e87]
[2] = /lib/x86_64-linux-gnu/libc.so.6(abort+0x141) [0x7f442283a7f1]
[3] = /lib/x86_64-linux-gnu/libc.so.6(+0x303fa) [0x7f442282a3fa]
[4] = /lib/x86_64-linux-gnu/libc.so.6(+0x30472) [0x7f442282a472]
[5] = /root/miniconda3/envs/legion-xxx/bin/legion_python(+0x126c69f) [0x5608babb469f]
[6] = /root/miniconda3/envs/legion-xxx/bin/legion_python(+0xdd7723) [0x5608ba71f723]
[7] = /root/miniconda3/envs/legion-xxx/bin/legion_python(+0xdd7906) [0x5608ba71f906]
[8] = /root/miniconda3/envs/legion-xxx/bin/legion_python(+0x127151a) [0x5608babb951a]
[9] = /root/miniconda3/envs/legion-xxx/bin/legion_python(+0xdda6a2) [0x5608ba7226a2]
[10] = /root/miniconda3/envs/legion-xxx/bin/legion_python(+0x1270766) [0x5608babb8766]
[11] = /root/miniconda3/envs/legion-xxx/bin/legion_python(+0xddce3b) [0x5608ba724e3b]
[12] = /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f44236fe6db]
[13] = /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f442291b61f]

[manopapad]

Assertion ‘nanoseconds <= uint64_t(LLONG_MAX)' failed.

This looks like a Realm issue (or rather, the output of some system timing function is not as Realm expects). If you have time, could you add this instrumentation and try building (and running) again with -DCMAKE_BUILD_TYPE=Release?

diff --git a/runtime/realm/timers.inl b/runtime/realm/timers.inl
index 75a4b42dc..8949fb03d 100644
--- a/runtime/realm/timers.inl
+++ b/runtime/realm/timers.inl
@@ -69,6 +69,14 @@ namespace Realm {
       nanoseconds -= zero_time;
 #ifdef DEBUG_REALM
     // make sure this fits in a long long
+    std::cerr << "native = " << native
+              << " nanoseconds = " << nanoseconds
+              << " absolute = " << absolute
+              << " zero_time = " << zero_time
+              << " LLONG_MAX = " << LLONG_MAX
+              << " uint64_t(LLONG_MAX) = " << uint64_t(LLONG_MAX)
+              << std::endl;
+    std::cerr.flush();
     assert(nanoseconds <= uint64_t(LLONG_MAX));
 #endif
     return nanoseconds;
@@ -325,4 +333,3 @@ namespace Realm {


 }; // namespace Realm
-

After the legate is successfully installed, run the legate command. The following error information is displayed:

Can you please share the full build commands that you used, and build output that you got, for Legion and legate.core?

[1193749292]

Add std::cerr... and with -DCMAKE_BUILD_TYPE=Release , the error information is the same.

The following error information is displayed without -DCMAKE_BUILD_TYPE=Release :

native = 125710472513989694 nanoseconds = 1645055 absolute = 0 zero_time = 1681439899479082722 LLONG_MAX = 9223372036854775807 uint64_t(LLONG_MAX) = 9223372036854775807
native = 125710472514051510 nanoseconds = 1665654 absolute = 0 zero_time = 1681439899479082722 LLONG_MAX = 9223372036854775807 uint64_t(LLONG_MAX) = 9223372036854775807
...(Several similar messages)

Signal 6legion_python: /root/xxx/legion-control_replication/runtime/realm/timers.inl:80: static long long int Realm::Clock::current_time_in_nanoseconds(bool): Assertion `nanoseconds <= uint64_t(LLONG_MAX)' failed.
 uint64_t(LLONG_MAX) = 9223372036854775807
9223372036854775807
legion_python: /root/xxx/legion-control_replication/runtime/realm/timers.inl:80: static long long int Realm::Clock::current_time_in_nanoseconds(bool): Assertion `nanoseconds <= uint64_t(LLONG_MAX)' failed.
 received by node 0, process 

[manopapad]

I reported the timing issue on StanfordLegion/realm#180. You can probably continue using a release build while the issue is being fixed. Thanks for your help investigating this.

Please share the command line you use to build in release mode, and the build outputs, so we can fix any remaining issues with that. And please make sure you include all the necessary Legion defines, when building Legion:

-DLegion_REDOP_HALF=ON -DLegion_REDOP_COMPLEX=ON -DLegion_BUILD_BINDINGS=ON -DLegion_USE_Python=ON

[1193749292]

@manopapad

cmake .. -S /root/xxx/legion-control_replication/ -D CMAKE_INSTALL_PREFIX=../install -D REALM_USE_OPENMP=ON -D Legion_USE_OpenMP=ON -D REALM_OPENMP_SYSTEM_RUNTIME=ON -D Legion_OpenMP_SYSTEM_RUNTIME=ON -D REALM_USE_MPI=ON -DLegion_NETWORKS=mpi -D Legion_MPI_INTEROP=ON -DLegion_REDOP_HALF=ON -DLegion_REDOP_COMPLEX=ON -DLegion_BUILD_BINDINGS=ON -DLegion_USE_Python=ON -DLegion_BUILD_JUPYTER=ON -DCMAKE_BUILD_TYPE=Release -B ../build
legion_build_output-success.txt

./install.py -j 40 --clean --network mpi --openmp --with-legion /root/xxx/legion-control_replication/build/ -v --nocuda
legate.core-install_output-success.zip

legate
legate_run-error.txt

https://gitlab.com/StanfordLegion/legion/-/commit/78c5f3ab9c4ea1393183ca135fbca3c403a3113b
Other than that, there are no other changes to the source code.

legion control_replication
legate branch-22.12