Windows CUDA release version missing

[fuzzballb]

The link to the windows CUDA version in the releases is giving a 404

https://github.com/PrismML-Eng/llama.cpp/releases/download/prism-b8846-d104cf1/llama-prism-b8846-d104cf1-bin-win-cuda-12.4-x64.zip

[fuzzballb]

I build it from source and it works perfectly

Running Ternary Bonsai with CUDA on Windows

Requirements: Windows, NVIDIA GPU (Maxwell/GTX 900 series or newer), ~5 GB free disk space

STEP 1 - Install prerequisites

• Visual Studio 2022 Community: https://visualstudio.microsoft.com/
  During install, select "Desktop development with C++" workload

• CMake: https://cmake.org/download/

• Git: https://git-scm.com/

• CUDA Toolkit 12.4: https://developer.nvidia.com/cuda-12-4-1-download-archive?target_os=Windows&target_arch=x86_64&target_version=11&target_type=exe_network
  During install, select Custom and UNCHECK the driver — keep only the Toolkit

STEP 2 - Download models

Get whichever .gguf you want from PrismML's Hugging Face page:
Ternary-Bonsai-1.7B-Q2_0.gguf (~0.4 GB) — fastest
Ternary-Bonsai-4B-Q2_0.gguf (~0.9 GB) — best balance

Place them somewhere easy, e.g. C:\Ternary\

STEP 3 - Download CUDA runtime DLLs

From https://github.com/PrismML-Eng/llama.cpp/releases/tag/prism-b8846-d104cf1
download: cudart-llama-bin-win-cuda-12.4-x64.zip
Extract it to C:\Ternary\cudart\

STEP 4 - Build llama.cpp from source

Open: Start Menu -> "Developer Command Prompt for VS 2022"

Run these commands:

cd C:\Ternary
git clone https://github.com/PrismML-Eng/llama.cpp.git llama.cpp-src
cd llama.cpp-src
set CUDA_PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4
cmake -B build -G Ninja -DGGML_CUDA=ON -DCMAKE_CUDA_COMPILER="%CUDA_PATH%\bin\nvcc.exe" -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j 4

This will take 10-20 minutes.

STEP 5 - Copy DLLs

Copy these three files from C:\Ternary\cudart\ into C:\Ternary\llama.cpp-src\build\bin:
cublas64_12.dll
cublasLt64_12.dll
cudart64_12.dll

STEP 6 - Run

C:\Ternary\llama.cpp-src\build\bin\llama-cli.exe -m C:\Ternary\Ternary-Bonsai-4B-Q2_0.gguf -c 2048 -n -1 --conversation -ngl 99 -t 4

Expected speeds (GTX 930MX, 2 GB VRAM):
1.7B model -> ~15 tokens/sec
4B model -> ~7 tokens/sec

Faster GPUs with more VRAM will do significantly better.
The 4B model is recommended as the best balance of speed and quality.

[vinodkrishna221]

i am also having this same issue

[khosravipasha]

hmm seems windows has the wrong link, I see the artifact in the list, Need to see what went wrong the release is automated with a github action

https://github.com/PrismML-Eng/llama.cpp/releases/download/prism-b8846-d104cf1/cudart-llama-bin-win-cuda-12.4-x64.zip

https://github.com/PrismML-Eng/llama.cpp/releases/download/prism-b8846-d104cf1/llama-prism-b1-d104cf1-bin-win-cuda-12.4-x64.zip

[copilot-swe-agent]

The agent encountered an error and was unable to start working on this issue: The service is temporarily unavailable. Please try again later. (Request id: D3F8:9E1E6:15AC77:15D7B5:69E8F4EF)

[khosravipasha]

On the https://github.com/PrismML-Eng/Bonsai-demo this is handled, not sure why windows one gets the wrong name.
Need to take a closer look, in the meantime can use the windows installer there (should handle everything automatically)
https://github.com/PrismML-Eng/Bonsai-demo/blob/main/setup.ps1