feat: builtin ROCm support

[ExposedCat]

Feature Description

I can't reproduce my existing AMD hipBLAS build:

export ROCM_PATH=/usr/lib64/rocm
export ROCM_DEVICE_LIB_PATH=$ROCM_PATH/llvm/lib/clang/18/amdgcn/bitcode
cmake -S . -B build \
  -DGGML_HIP=ON \
  -DAMDGPU_TARGETS=gfx1030 \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_C_COMPILER=hipcc \
  -DCMAKE_CXX_COMPILER=hipcc
cmake --build build --config Release -j$(nproc)

error:

    sh: line 1: /usr/lib64/rocm/bin/rocm_agent_enumerator: No such file or directory
    clang++: error: cannot find ROCm device library; provide its path via '--rocm-path' or '--rocm-device-lib-path', or pass '-nogpulib' to build without ROCm device library

The Solution

Allow to use existing llamacpp binaries build

Considered Alternatives

Tried getLlama options and GGML options for building from source, none worked

Additional Context

No response

Related Features to This Feature Request

• Metal support
• CUDA support
• Vulkan support
• Grammar
• Function calling

Are you willing to resolve this issue by submitting a Pull Request?

Might take a look, but have no experience with ROCm

[giladgd]

@ExposedCat node-llama-cpp has its own bindings to llama.cpp, meaning it compiles custom binaries with an interface created specifically for consumption from nodejs by node-llama-cpp; thus you cannot use any binaries built outside this library as they're incompatible.

You can try to set relevant cmakeOptions on getLlama and and configure custom env vars (by setting them on process.env if needed), together with gpu: false to try and build with ROCm support.
From the build instructions that you shared, setting the relevant env vars might be the missing piece here:

process.env["ROCM_PATH"] = "/usr/lib64/rocm";
process.env["ROCM_DEVICE_LIB_PATH"] = process.env["ROCM_PATH"] + "/llvm/lib/clang/18/amdgcn/bitcode";

const llama = await getLlama({
    gpu: false,
    existingPrebuiltBinaryMustMatchBuildOptions: true,
    cmakeOptions: {
        "GGML_HIP": "ON",
        "AMDGPU_TARGETS": "gfx1030",
        "CMAKE_C_COMPILER": "hipcc",
        "CMAKE_CXX_COMPILER": "hipcc"
    }
});

delete process.env["ROCM_PATH"];
delete process.env["ROCM_DEVICE_LIB_PATH"];

I don't have a machine with an AMD GPU nor have access to one, so I can't add official support for ROCm myself and maintain support for it.
From my tests, the performance of Vulkan is comparable to the performance of CUDA on the machines that I tested both on, so given that AMD GPUs have decent Vulkan support I suppose the performance of Vulkan would be comparable to ROCm on such machines, but it's worth testing.

If you're interested in adding official support for ROCm in node-llama-cpp yourself and open a PR for it, you can take a look at my guidance for adding SYCL support, as adding support for ROCm should be very similar.
Let me know if you need any additional help, or if you get stuck on something.