make bpftools build fails: `ld.lld: error: unable to find library -lLLVM` when linking `bpftrace-aotrt`

[alfonsoalongi]

Steps to reproduce
Execute standard steps:

# Build the Docker image
./scripts/build-docker-image.sh
# Run the environment
./scripts/run-docker-build-env.sh
# clean the project
make clean
# Create the artifact in out/archives/bpftools-arm64.tar.gz
make bpftools THREADS=$(nproc)

Description
Following the steps, the build stops at 96% during the bpftrace step with error "unable to find library -lLLVM" during the phase "Linking CXX executable bpftrace-aotrt". The same issue also occurs if I use make bpftools-min.
Can you fix this issue?

uname  -r
5.14.0-503.40.1.el9_5.x86_64
podmav -v
podman version 5.2.2

Thank you for the awesome project and thanks in advance for for your reply.

** Logs**

make[3]: Leaving directory '/ExtendedAndroidTools/build/android/arm64/bpftrace'
[ 64%] Built target runtime
[ 87%] Built target ast
make[3]: Entering directory '/ExtendedAndroidTools/build/android/arm64/bpftrace'
make[3]: Entering directory '/ExtendedAndroidTools/build/android/arm64/bpftrace'
make[3]: Leaving directory '/ExtendedAndroidTools/build/android/arm64/bpftrace'
Consolidate compiler generated dependencies of target libbpftrace
make[3]: Entering directory '/ExtendedAndroidTools/build/android/arm64/bpftrace'
**[ 88%] Linking CXX executable bpftrace-aotrt
make[3]: Leaving directory '/ExtendedAndroidTools/build/android/arm64/bpftrace'
ld.lld: error: unable to find library -lLLVM
clang++: error: linker command failed with exit code 1 (use -v to see invocation)**
make[3]: *** [src/aot/CMakeFiles/bpftrace-aotrt.dir/build.make:112: src/aot/bpftrace-aotrt] Error 1
make[3]: Leaving directory '/ExtendedAndroidTools/build/android/arm64/bpftrace'
make[2]: *** [CMakeFiles/Makefile2:638: src/aot/CMakeFiles/bpftrace-aotrt.dir/all] Error 2
make[2]: *** Waiting for unfinished jobs....
make[3]: Entering directory '/ExtendedAndroidTools/build/android/arm64/bpftrace'
[ 89%] Building CXX object src/CMakeFiles/libbpftrace.dir/clang_parser.cpp.o
/ExtendedAndroidTools/projects/bpftrace/sources/src/clang_parser.cpp:387:25: warning: moving a temporary object prevents copy elision [-Wpessimizing-move]
  387 |             enum_name = std::move(name.str());
      |                         ^
/ExtendedAndroidTools/projects/bpftrace/sources/src/clang_parser.cpp:387:25: note: remove std::move call here
  387 |             enum_name = std::move(name.str());
      |                         ^~~~~~~~~~          ~
1 warning generated.
[ 91%] Linking CXX static library libbpftrace.a
make[3]: Leaving directory '/ExtendedAndroidTools/build/android/arm64/bpftrace'
[ 96%] Built target libbpftrace
make[2]: Leaving directory '/ExtendedAndroidTools/build/android/arm64/bpftrace'
make[1]: *** [Makefile:136: all] Error 2
make[1]: Leaving directory '/ExtendedAndroidTools/build/android/arm64/bpftrace'
make: *** [projects/bpftrace/build.mk:25: build/android/arm64/bpftrace.done] Error 2

[4399erbai]

I guess it's because you didn't use android-ndk-r27b to build. I had the same problem when using android-ndk-r23b or r25b.

[alfonsoalongi]

Hi @4399erbai,

The scripts/download-ndk.sh script automatically downloads android-ndk-r27b, so the build should never fall back to r23b/r25b.

Quick check:

./scripts/run-docker-build-env.sh   # start the container
ls -al /opt/ndk                     # inside the container

drwxr-xr-x. 11 root root 4096 Aug 29  2024 android-ndk-r27b

---

The repository has changed

The codebase has been updated since the issue was opened, and I now get a different error regardless of whether I set NDK_DIR.

Note: When I first opened the issue, I solved it using the procedure in the README
👉 https://github.com/alfonsoalongi/bpftools-builder/blob/main/README.md

---

Current build log

export NDK_DIR=/opt/ndk/android-ndk-r27b
make bpftools-min NDK_ARCH=arm64 THREADS=$(nproc)

First run

...
cd build/host/llvm && /ExtendedAndroidTools/out/host/bin/cmake /ExtendedAndroidTools/projects/llvm/sources/llvm \
        -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra" -DLLVM_TARGETS_TO_BUILD="AArch64;BPF" \
        -DLLVM_TEMPORARILY_ALLOW_OLD_TOOLCHAIN=1 \
        -DCMAKE_BUILD_TYPE=Release
CMake Error: The source directory "/ExtendedAndroidTools/projects/llvm/sources/llvm" does not exist.
Specify --help for usage, or press the help button on the CMake GUI.
make: *** [projects/llvm/build.mk:74: build/host/llvm] Error 1

Second run

...
cd build/host/llvm && make -j 20 clang-tblgen
make[1]: *** No rule to make target 'clang-tblgen'.  Stop.
make[1]: Leaving directory '/ExtendedAndroidTools/build/host/llvm'
make: *** [projects/llvm/build.mk:68: out/host/bin/clang-tblgen] Error 2

It looks like the LLVM clone is being interrupted, leaving the build directory incomplete and causing the subsequent clang-tblgen failure.

---

I’m investigating this, as I don’t remember whether the same issue appeared in my CI workflow for releasing the artifacts (https://github.com/alfonsoalongi/bpftools-builder/blob/main/.github/workflows/bpftools-release.yml). However, the latest workflow currently works.

In any case, with the previous commit, the approach in the README still works (first step: build LLVM + Clang for the host). I haven’t tested it with the new version yet.