Dear all,
Long time listener/reader, first time poster.
I’m wondering if anybody has a working dockerfile to get CUDA ik_llama.cpp compiling.
Since ik_llama has returned to GitHub, I’ve cloned it and was planning on trying out the ubergarm Q4 quant of Qwen3 235B 2507.
There dockerfiles in .devops are different to the parent llama.cpp, but I haven’t successfully gotten my docker build to compile yet.
The initial error was
68.51 ggml/src/ggml-cuda/convert.cu(358): error: more than one conversion function from “const half” to a built-in type applies:
This it seems is related to trying to compile for ARG CUDA_DOCKER_ARCH=“all”
Things I have tried:
CUDA_VERSION bump to latest 12.9.1 (>11.7 to allow for Ada compilation)
CUDA_DOCKER_ARCH=“sm_86 sm_89” (I have 3090 and 4090x2)
This got me further, to run into requiring small make flag changes
RUN make NVCCFLAGS=“-DGGML_CUDA_MIN_BATCH_OFFLOAD=32 --extended-lambda”. During troubleshooting, I shifted the gpu architecture specification to the make line with -arch=sm_86 -arch=sm_89 (removing the environment variable); and then reduced to the lowest common denominator -arch=sm_86 in case multiple architectures was an issue
Every compilation step came with warnings about an incompatible redefinition of gpu-architecture in nvcc, but since compilation progressed it appears sm_86 was used as requested.
After all of this, compilation would progress reasonably far through all of ggml_cuda/template-instances, and on to this
_cuda_flash_attn_ext_mma_f16_case<128, 2, 8>(ggml_backend_cuda_context&, ggml_tensor*)’
1470.5 /usr/bin/ld: tmpxft_00000197_00000000-6_fattn.cudafe1.cpp:(.text+0x1108e): undefined reference to void ggml_cuda_flash_attn_ext_mma_f16_case<80, 8, 4>(ggml_backend_cuda_context&, ggml_tensor*)' 1470.5 /usr/bin/ld: tmpxft_00000197_00000000-6_fattn.cudafe1.cpp:(.text+0x110a1): undefined reference to void ggml_cuda_flash_attn_ext_mma_f16_case<80, 4, 4>(ggml_backend_cuda_context&, ggml_tensor*)’
1470.5 /usr/bin/ld: tmpxft_00000197_00000000-6_fattn.cudafe1.cpp:(.text+0x110b4): undefined reference to `void ggml_cuda_flash_attn_ext_mma_f16_case<80, 2, 4>(ggml_backend_cuda_context&, ggml_tensor*)’
1470.8 collect2: error: ld returned 1 exit status
1470.8 make: *** [Makefile:1380: llama-baby-llama] Error 1
full-cuda.Dockerfile:37
35 | ENV GGML_RPC=1
36 |
37 | >>> RUN make NVCCFLAGS=“-DGGML_CUDA_MIN_BATCH_OFFLOAD=32 --extended-lambda -arch=sm_86”
Where I get about 30 similar errors for case<64/80/96/112/128/256 etc
Can anybody provide any insight into the issue and how to solve it?
Thank you in advance!