[Bug]在使用最新的latest-cu128的docker 镜像的时候出现 illegal memory access

[simonjhy]

Checklist

• 1. I have searched related issues but cannot get the expected help.
• 2. The bug has not been fixed in the latest version.
• 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

Describe the bug

在使用openmmlab/lmdeploy:latest-cu12.8和openmmlab/lmdeploy:latest-cu12的时候，输入任何内容都会触发下面的错误。使用的环境是V100 32GB * 8.

[TM][ERROR] CUDA runtime error: an illegal memory access was encountered /opt/lmdeploy/src/turbomind/core/stream.h:27

Reproduction

下载最新版的docker pull openmmlab/lmdeploy:latest-cu12.8
加载glm-4.7-flash-awq模型
执行推理任务的时候出现错误，会导致框架重新启动

Environment

lmdeploy:latest-cu12.8
ubuntu 2204

Error traceback

[TM][ERROR] CUDA runtime error: an illegal memory access was encountered /opt/lmdeploy/src/turbomind/core/stream.h:27
[TM][ERROR] CUDA runtime error: an illegal memory access was encountered /opt/lmdeploy/src/turbomind/core/allocator.cc:49
[TM][ERROR] CUDA runtime error: an illegal memory access was encountered /opt/lmdeploy/src/turbomind/core/allocator.cc:49
[TM][ERROR] void turbomind::LlamaLinear::Impl::Forward(turbomind::core::Tensor&, const turbomind::core::Tensor&, const turbomind::LlamaDenseWeight&, const turbomind::core::Buffer_<int>&, const turbomind::core::Buffer_<int>&): 1
[TM][ERROR] CUDA runtime error: an illegal memory access was encountered /opt/lmdeploy/src/turbomind/core/allocator.cc:55

[simonjhy]

任何一个请求都会触发这个错误

[simonjhy]

这个真需要快点修复啊，谢谢

[simonjhy]

使用了你们最新发布的dev-cu128版本的docker image之后，还是有CUDA runtime error: an illegal memory access的异常，下面的是具体的log。
INFO: 172.19.0.4:51406 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[TM][INFO] [SeqMgr][Create] ID 1
[TM][INFO] [SeqMgr][match] ID 1, hit blocks 0, cache_len 0
[TM][INFO] [SeqMgr][match] ID 1, after matching, blocks 0, cache_len 0
[TM][INFO] [BlockTrie][cache] 8192 new tokens cached
[TM][INFO] [BlockTrie][cache] 8192 new tokens cached
[TM][INFO] [BlockTrie][cache] 8192 new tokens cached
[TM][INFO] [BlockTrie][cache] 8192 new tokens cached
[TM][INFO] [SeqMgr][CachePrompt] ID 1, cached blocks 1024, tokens 13551
[TM][INFO] [BlockTrie][cache] 5352 new tokens cached
[TM][INFO] [BlockTrie][cache] 5352 new tokens cached
[TM][INFO] [SeqMgr][CachePrompt] ID 1, cached blocks 1693, tokens 13551
[TM][INFO] [BlockTrie][cache] 5352 new tokens cached
[TM][INFO] [BlockTrie][cache] 5352 new tokens cached
[TM][ERROR] CUDA runtime error: an illegal memory access was encountered /opt/lmdeploy/src/turbomind/core/allocator.cc:55

[lvhan028]

glm-4.7-flash-awq 的模型链接是这个么？https://huggingface.co/QuantTrio/GLM-4.7-Flash-AWQ

[simonjhy]

对，我使用的就是这个版本的，我几乎用了你们发布的所有版本的cu128的image，latest，dev的，还尝试了latest-cu12的，都不行。之前我用的是大概一周前的latest-cu128版本的image，总是报INPUT_LENGTH_ERROR这个错误。昨天我更新到dev-cu128版本之后，就是只要是输入任何问题，无论长短全部是这个错误了
“[TM][ERROR] CUDA runtime error: an illegal memory access was encountered /opt/lmdeploy/src/turbomind/core/allocator.cc:55”

LMDeploy是难得现在还能对V100提供很好加速支持的框架，真的非常感激和感动，国内开源的优秀的框架。

[simonjhy]

environment:
  CUDA_VISIBLE_DEVICES: "0,1,2,3"
  CUDA_DEVICE_ORDER: "PCI_BUS_ID"
  NCCL_DEBUG: "WARN"
  NCCL_P2P_LEVEL: "NVL"
  NCCL_P2P_DISABLE: "0"
  NCCL_SHM_DISABLE: "0"
  NCCL_IB_DISABLE: "0"
  NCCL_SOCKET_IFNAME: "eth0"
  NCCL_TIMEOUT: "1800"
  NCCL_NVLS_ENABLE: "0"
  CUDA_LAUNCH_BLOCKING: "0"
  CUDA_DEVICE_MAX_CONNECTIONS: "32"
  OMP_NUM_THREADS: "8"
  OMP_PROC_BIND: "close"
  OMP_PLACES: "cores"
  TOKENIZERS_PARALLELISM: "false"
ipc: host
shm_size: "32gb"
cpuset: "0-23,48-71"
ulimits:
  memlock:
    soft: -1
    hard: -1
  nofile:
    soft: 65536
    hard: 65536
ports:
  - "9991:8000"
volumes:
  - /data/ai/models/vllm:/models
  - /tmp/lmdeploy_cache_1:/root/.cache
command: >
  lmdeploy serve api_server
  /models/GLM-4.7-Flash-AWQ
  --model-name GLM_4.7_30B
  --server-name 0.0.0.0
  --server-port 8000
  --backend turbomind
  --tp 4
  --session-len 102400
  --cache-max-entry-count 0.9
  --cache-block-seq-len 8
  --max-batch-size 1
  --quant-policy 4
  --enable-prefix-caching
  --max-concurrent-requests 6
  --max-prefill-token-num 8192
  --max-prefill-iters 12
  --rope-scaling-factor 0.0
  --log-level INFO
  --communicator nccl
  --tool-call-parser qwen3

我是用这个配置，只要执行就报CUDA runtime error: an illegal memory acces。我这个配置有错误？

[simonjhy]

environment: CUDA_VISIBLE_DEVICES: "0,1,2,3" CUDA_DEVICE_ORDER: "PCI_BUS_ID" NCCL_DEBUG: "WARN" NCCL_P2P_LEVEL: "NVL" NCCL_P2P_DISABLE: "0" NCCL_SHM_DISABLE: "0" NCCL_IB_DISABLE: "0" NCCL_SOCKET_IFNAME: "eth0" NCCL_TIMEOUT: "1800" NCCL_NVLS_ENABLE: "0" CUDA_LAUNCH_BLOCKING: "0" CUDA_DEVICE_MAX_CONNECTIONS: "32" OMP_NUM_THREADS: "8" OMP_PROC_BIND: "close" OMP_PLACES: "cores" TOKENIZERS_PARALLELISM: "false" ipc: host shm_size: "32gb" cpuset: "0-23,48-71" ulimits: memlock: soft: -1 hard: -1 nofile: soft: 65536 hard: 65536 ports: - "9991:8000" volumes: - /data/ai/models/vllm:/models - /tmp/lmdeploy_cache_1:/root/.cache command: > lmdeploy serve api_server /models/GLM-4.7-Flash-AWQ --model-name GLM_4.7_30B --server-name 0.0.0.0 --server-port 8000 --backend turbomind --tp 4 --session-len 102400 --cache-max-entry-count 0.9 --cache-block-seq-len 8 --max-batch-size 1 --quant-policy 4 --enable-prefix-caching --max-concurrent-requests 6 --max-prefill-token-num 8192 --max-prefill-iters 12 --rope-scaling-factor 0.0 --log-level INFO --communicator nccl --tool-call-parser qwen3 我是用这个配置，只要执行就报CUDA runtime error: an illegal memory acces。我这个配置有错误？ Lyu Han ***@***.***> 于2026年3月17日周二 18:53写道：

…

*lvhan028* left a comment (InternLM/lmdeploy#4415) <#4415 (comment)> 我在 v100 上用 QuantTrio/GLM-4.7-Flash-AWQ <https://huggingface.co/QuantTrio/GLM-4.7-Flash-AWQ> 测试没问题。 这是服务命令： lmdeploy serve api_server /workspace/models/huggingface_hub/hub/models--QuantTrio--GLM-4.7-Flash-AWQ/snapshots/88e3d3d913c0d97c8f505cdc03433c48226bedc3/ --tp 2 --backend turbomind --log-level INFO — Reply to this email directly, view it on GitHub <#4415 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABL2WJAXH24EILYTIWW3FDT4REVC5AVCNFSM6AAAAACWS7KHOOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DANZUGA3DSOJXHA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- aquila

[lvhan028]

我这边测试正常，使用的模型是 https://huggingface.co/QuantTrio/GLM-4.7-Flash-AWQ
部署命令是：

 lmdeploy serve api_server /workspace/models/huggingface_hub/hub/models--QuantTrio--GLM-4.7-Flash-AWQ/snapshots/88e3d3d913c0d97c8f505cdc03433c48226bedc3/ --tp 2 --backend turbomind --log-level INFO

镜像是 openmmlab/lmdeploy:latest-cu12.8

[lvhan028]

不要使用：--quant-policy 4

[simonjhy]

好的，谢谢。问一下，在推理的时候发现产生这样的error是什么错误？上下文超长，还是超时引起的框架异常？

2026-03-18 05:50:47,381 - lmdeploy - ERROR - async_engine.py:271 - [safe_run] session 1 exception caught: CancelledError Cancelled via cancel scope 77b99c194620 by <Task pending name='Task-13' coro=<RequestResponseCycle.run_asgi() running at /opt/py3/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py:410> cb=[set.discard()]>

[lvhan028]

这个模型不支持 --quant-policy 4，也就是对 kv cache 进行 4bit 量化。
"lmdeploy serve api_server --help" 可以查看具体信息

[jingyibo123]

@lvhan028
请问GLM 4.7 Flash W4A16 AWQ量化 Turbomind有支持 KV cache量化的计划吗？
另外该模型的tool-parser和reasoing-parser没看到进行中的PR？