[None][docs] Update supported models matrix with AD-onboarded architectures

[bmarimuthu-nv]

Summary

• Add 15 new architecture entries to the model support matrix for models onboarded via AutoDeploy
• Expand 8 existing architecture entries with broader model family coverage
• Add [^7] footnote for all AutoDeploy-supported architectures

New architectures

DeepseekV2ForCausalLM, ExaoneForCausalLM, Gemma2ForCausalLM, GemmaForCausalLM, GlmMoeDsaForCausalLM, GraniteMoeHybridForCausalLM, HunYuanDenseV1ForCausalLM, HunYuanMoEV1ForCausalLM, InternLM2ForCausalLM, Olmo2ForCausalLM, OpenELMForCausalLM, Phi4FlashForCausalLM, Phi4VisionRForConditionalGeneration, SeedOssForCausalLM, Starcoder2ForCausalLM

Test plan

• Verify markdown renders correctly
• Cross-check architecture names against AD custom models __init__.py

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Added support for 30+ new language model architectures (Qwen3.5, Phi-4, HunYuan, Llama-3/4, Mistral-3, and more) for AutoDeploy export.
  • Introduced registry-based model configuration system enabling centralized YAML management.
  • Added agent-based workflows for automated model onboarding, execution, and debugging.
  • Expanded supported models table in documentation.

• Infrastructure

  • Enhanced AutoDeploy custom operators including Mamba/ySamba SSM support.
  • Updated default runtime settings (TensorRT-LLM backend, torch-cudagraph compiler).

[coderabbitai]

📝 Walkthrough

Walkthrough

This pull request introduces a comprehensive AutoDeploy infrastructure expansion, including agent definitions and configurations for model onboarding workflows, new AutoDeploy model implementations for 30+ architectures, model registry configurations, custom operator enhancements for Mamba/SSM and attention backends, and runtime framework updates for registry-based model configuration management.

Changes

Cohort / File(s)	Summary
Agent & Workflow Infrastructure .claude/agents/ad-*.md, .codex/AGENTS.md, .codex/agents/ad_*.toml, .codex/agents/onboard_*.toml, .codex/config.toml	New agent definitions and configurations for AutoDeploy debugging, model review, run execution, and onboarding progress tracking with detailed developer instructions, checklist templates, and sandbox/reasoning effort settings.
Documentation & Skill Guides .claude/skills/ad-model-onboard/SKILL.md, .codex/skills/ad-model-onboard/SKILL.md, AGENTS.md	Expanded AutoDeploy onboarding skill documentation with 11-phase workflow including GPU checks, registry integration, canonical ops requirements, and PR procedures; added GitHub CLI authentication guidance.
Model Registry Configuration examples/auto_deploy/model_registry/models.yaml, examples/auto_deploy/model_registry/configs/*.yaml, examples/auto_deploy/model_registry/generate_csv.py	Registry entries for 30+ model architectures with per-model YAML overrides for backends, transforms, and sharding; CSV generation script for model inventory; removed dashboard defaults and added specialized configs for Granite, Qwen, Phi, HunYuan, DeepSeek variants.
Custom Operators: Attention & MoE tensorrt_llm/_torch/auto_deploy/custom_ops/attention/trtllm_attention.py, tensorrt_llm/_torch/auto_deploy/custom_ops/attention_interface.py, tensorrt_llm/_torch/auto_deploy/custom_ops/linear/torch_router.py	Updates to TensorRT-LLM attention handler with dynamic max_blocks_per_seq, attention window sizing fixes, scaling adjustments; made max_num_tokens required in SequenceInfo; fixed router device placement for fake implementation.
Custom Operators: Mamba/SSM tensorrt_llm/_torch/auto_deploy/custom_ops/mamba/torch_mamba.py, tensorrt_llm/_torch/auto_deploy/custom_ops/mamba/torch_backend_mamba.py	New ySamba SSM operators with prefill and cached decode paths, including state management, decode iteration logic, and backend registration; prefill implementation with per-token discretization and state updates.
Custom Operators: FLA/Gating tensorrt_llm/_torch/auto_deploy/custom_ops/fla/fla_gated_delta.py, tensorrt_llm/_torch/auto_deploy/custom_ops/fla/fla_backend_gated_delta.py, tensorrt_llm/_torch/auto_deploy/custom_ops/fla/torch_backend_gated_delta.py	GQA support for gated delta rule with separate K/V head tracking; cache sizing aligned with V-head count; repeat_interleave-based head expansion when num_v_heads > num_k_heads.
Custom Operators: Documentation tensorrt_llm/_torch/auto_deploy/custom_ops/README.md	Added ySamba and cached ySamba SSM operator entries to reference documentation.
Framework Configuration & Utilities tensorrt_llm/_torch/auto_deploy/config/default.yaml, tensorrt_llm/_torch/auto_deploy/llm_args.py	Changed default backends from flashinfer/torch-compile to trtllm/torch-cudagraph; increased max_seq_len (512→2048) and max_batch_size (8→64); adjusted CUDA graph batch-size generation heuristic.
Runtime Modifications examples/auto_deploy/build_and_run_ad.py	Added registry-based YAML resolution via --use-registry flag; refactored prompt handling to remove benchmark logic; added query preparation with chat template detection; changed batch_size default from 2 to 10.
Model Loading & Multimodal tensorrt_llm/_torch/auto_deploy/llm.py	Multimodal message normalization for processors with image_processor; token_type_ids removal from forward args to prevent extra argument passing.
Model Implementations (Dense/Decoder) tensorrt_llm/_torch/auto_deploy/models/custom/modeling_{llama3,llama4,mistral,mistral3,cohere,decilm,deepseek_v2,exaone}.py	New prefill-only causal LM implementations for Llama3/4, Mistral, Mistral3 (with multimodal), Cohere, DeciLM, DeepSeek-V2 (with MoE/MLA), Exaone using canonical ops for RoPE, attention, and MLP.
Model Implementations (Specialized) tensorrt_llm/_torch/auto_deploy/models/custom/modeling_{gemma,gemma2,gemma3,openelm,phi4,phi4flash,phi4mm,phi4_visionr}.py	Gemma/Gemma2/Gemma3 variants with scaling and softcapping; OpenELM with per-layer MLP sizing; Phi-4 standard/Flash/multimodal/vision implementations with RoPE and MLP variants.
Model Implementations (MoE/Hybrid) tensorrt_llm/_torch/auto_deploy/models/custom/modeling_{glm4_moe,glm_moe_dsa,gpt_oss,granite_moe_hybrid,hunyuan_dense,hunyuan_moe,minimax_m2,qwen3_moe}.py	MoE variants for GLM4/GLM-MoE-DSA, GPT-OSS, Granite, HunYuan, MiniMax-M2, Qwen3 with expert routing, top-k selection, and shared experts via canonical ops.
Model Implementations (Linear Attention) tensorrt_llm/_torch/auto_deploy/models/custom/modeling_{granite,internlm3,olmo3,qwen3,qwen3_5,starcoder2,seedoss,other_variants}.py	Models leveraging linear attention (FLA gated delta), Mamba SSM, attention variants with optional sliding windows, and per-layer architecture selection (attention vs Mamba).
Model Registry & Exports tensorrt_llm/_torch/auto_deploy/models/custom/__init__.py, docs/source/models/supported-models.md	Expanded __all__ exports for 30+ new model classes; updated supported-models documentation to list new AutoDeploy-backed architectures with feature matrix.

Estimated code review effort

🎯 5 (Critical) | ⏱️ ~120 minutes

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

[coderabbitai]

Actionable comments posted: 18

Note

Due to the large number of review comments, Critical, Major severity comments were prioritized as inline comments.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (3)

tensorrt_llm/_torch/auto_deploy/llm_args.py (1)

1-6: ⚠️ Potential issue | 🟡 Minor

Missing NVIDIA copyright header.

This Python file is missing the required NVIDIA copyright header. Per coding guidelines, all TensorRT-LLM source files should contain an Apache 2.0 license header.

🛡️ Proposed fix to add copyright header

+# SPDX-FileCopyrightText: Copyright (c) 2022-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 from importlib.resources import files

As per coding guidelines: "All TensorRT-LLM source files should contain an NVIDIA copyright header with the year of the latest meaningful modification."

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tensorrt_llm/_torch/auto_deploy/llm_args.py` around lines 1 - 6, Add the
required NVIDIA Apache-2.0 copyright header at the top of this source file
(tensorrt_llm._torch.auto_deploy.llm_args) including "Copyright (c) YEAR, NVIDIA
CORPORATION" with the year set to the latest meaningful modification, followed
by the standard Apache License, Version 2.0 boilerplate; place it before any
imports so the header covers the entire file.

.claude/agents/ad-onboard-reviewer.md (2)

118-121: ⚠️ Potential issue | 🟡 Minor

The sample PASS output still teaches the old config policy.

Lines 120-121 show a locally-defined config and AutoConfig.register() as an unconditional PASS, which contradicts the stricter B2/B3 rules above. Please update the example so it only blesses that pattern when the config is genuinely unavailable.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In @.claude/agents/ad-onboard-reviewer.md around lines 118 - 121, Update the
example PASS output so it does not unconditionally endorse a locally-defined
config and AutoConfig.register(); specifically change the lines referencing
modeling_foo.py, FooConfig, and AutoConfig.register("foo", FooConfig,
exist_ok=True) so the sample only shows this pattern as PASS when the global
config is genuinely unavailable (e.g., indicate a conditional note or change to
FAIL otherwise). Locate the example block that lists "modeling_foo.py:15 —
FooConfig defined in file" and "AutoConfig.register(...)" and modify the text to
reflect the stricter policy: either mark the register line as
conditional/pass-when-unavailable or mark it as failing guidance unless an
external config cannot be found.

---

72-74: ⚠️ Potential issue | 🟠 Major

Align D2/D3 with the new slice-once RoPE guidance.

Lines 73-74 still require RoPE to return full tables and be sliced inside attention, but .codex/skills/ad-model-onboard/SKILL.md Lines 310-311 now require the opposite. Leaving both docs unchanged will make the reviewer disagree with the skill on otherwise-correct model implementations.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In @.claude/agents/ad-onboard-reviewer.md around lines 72 - 74, Update D2 and D3
to match the new "slice-once" RoPE policy: change D2 to state that RoPE.forward
(the RoPE class' forward method) must return tensors already sliced to the
requested sequence length (not the full cached table), and change D3 to
remove/negate any requirement that downstream attention (e.g., the attention
forward implementation that indexes cos/sin via position_ids or
cos[position_ids]) perform additional slicing — attention should consume the
already-sliced RoPE outputs. Reference the RoPE.forward symbol and the attention
forward/position_ids usage when making these edits.

🟡 Minor comments (12)

tensorrt_llm/_torch/auto_deploy/llm.py-49-59 (1)

49-59: ⚠️ Potential issue | 🟡 Minor

In-place mutation of caller's messages may cause side effects.

The code modifies msg["content"] in-place for each message. If the caller reuses the inputs["messages"] list after this call, they may unexpectedly see the transformed format instead of their original plain strings.

Consider creating a copy of the messages list before modification to avoid mutating the caller's data.

🛡️ Proposed fix to avoid in-place mutation

             messages = inputs["messages"]
             is_multimodal = hasattr(self.processor, "image_processor")
             if is_multimodal:
+                # Create a shallow copy to avoid mutating the caller's data
+                messages = [dict(msg) for msg in messages]
                 for msg in messages:
                     if isinstance(msg.get("content"), str):
                         msg["content"] = [{"type": "text", "text": msg["content"]}]

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tensorrt_llm/_torch/auto_deploy/llm.py` around lines 49 - 59, The current
loop mutates the caller's inputs["messages"] in-place when is_multimodal
(hasattr(self.processor, "image_processor")), which can leak formatted message
objects back to the caller; instead, make a local copy of the messages list and
each message dict before transforming content (e.g., create a new list like
local_messages = [msg.copy() for msg in inputs["messages"]] and work on
local_messages), update only the copied msg["content"] to the
[{"type":"text","text":...}] form for string contents, and then use
local_messages for downstream processing so inputs["messages"] remains
unchanged; refer to inputs["messages"], is_multimodal, self.processor, and
msg["content"] to locate where to apply this change.

examples/auto_deploy/model_registry/configs/granite_4.0_micro.yaml-2-2 (1)

2-2: ⚠️ Potential issue | 🟡 Minor

Typo in comment: "attn ackend" → "attn backend".

📝 Proposed fix

-# Note: Uses flashinfer attention backend (trtllm attn ackend produces garbled output)
+# Note: Uses flashinfer attention backend (trtllm attn backend produces garbled output)

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@examples/auto_deploy/model_registry/configs/granite_4.0_micro.yaml` at line
2, There's a typo in the comment string "# Note: Uses flashinfer attention
backend (trtllm attn ackend produces garbled output)" — change "attn ackend" to
"attn backend" so the comment reads "...(trtllm attn backend produces garbled
output)"; update the comment text in the YAML (look for the exact comment
containing "flashinfer attention backend" / "trtllm attn ackend") to correct the
spelling.

.claude/agents/ad-debug-agent.md-53-53 (1)

53-53: ⚠️ Potential issue | 🟡 Minor

Minor grammar: "user given" should be hyphenated.

Per static analysis hint, "user given" should be "user-given" when used as a compound adjective.

📝 Proposed fix

-Run the AD flow with the user given model-id using the below command.
+Run the AD flow with the user-given model-id using the below command.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In @.claude/agents/ad-debug-agent.md at line 53, Update the sentence "Run the AD
flow with the user given model-id using the below command." to use the compound
adjective form by replacing "user given" with "user-given" so it reads "Run the
AD flow with the user-given model-id using the below command."; ensure only that
hyphenation change is applied and no other wording is altered.

.claude/agents/ad-debug-agent.md-72-82 (1)

72-82: ⚠️ Potential issue | 🟡 Minor

Duplicate instruction and typo.

1. Line 72 has a typo: "use you your own tools" should be "use your own tools"
2. Lines 72 and 82 are duplicates — consider removing one

📝 Proposed fix

-Remember to use you your own tools - Read, Grep, Glob, Bash, Edit, Write
+Remember to use your own tools - Read, Grep, Glob, Bash, Edit, Write

And remove line 82 as it duplicates line 72.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In @.claude/agents/ad-debug-agent.md around lines 72 - 82, Fix the duplicated
instruction and typo: change the phrase "Remember to use you your own tools -
Read, Grep, Glob, Bash, Edit, Write" to "Remember to use your own tools - Read,
Grep, Glob, Bash, Edit, Write" and remove the duplicate occurrence of that exact
line (the second instance) so it appears only once in the document; search for
the exact sentence to locate both instances to update and delete the duplicate.

.codex/agents/ad_onboard_reviewer.toml-63-63 (1)

63-63: ⚠️ Potential issue | 🟡 Minor

Checklist item G5 appears to be missing.

The checklist jumps from G4 to G6 in the "Test File Hierarchical Levels" section. If G5 was intentionally removed, consider renumbering G6 to G5 for consistency. If it was accidentally omitted, please add the missing item.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In @.codex/agents/ad_onboard_reviewer.toml at line 63, The checklist numbering
in the "Test File Hierarchical Levels" section skips G5 (it goes from G4 to G6);
either restore the missing G5 entry or renumber G6 to G5 for consistency. Locate
the checklist lines referencing G4 and G6 in
.codex/agents/ad_onboard_reviewer.toml and either insert the intended G5
checklist item between G4 and G6 or change the label "G6: Export test runs a
second forward with a different shape to validate dynamic dims." to "G5: ..." so
the sequence is contiguous.

examples/auto_deploy/model_registry/generate_csv.py-1-6 (1)

1-6: ⚠️ Potential issue | 🟡 Minor

Missing NVIDIA copyright header.

New Python files require the NVIDIA Apache 2.0 copyright header. Please add the standard header at the top of the file.

📝 Add copyright header

+# SPDX-FileCopyrightText: Copyright (c) 2022-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 """Generate a CSV from models.yaml with HF model id, link, and build_and_run_ad.py command."""

 import csv

As per coding guidelines: "All TensorRT-LLM source files should contain an NVIDIA copyright header with the year of the latest meaningful modification."

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@examples/auto_deploy/model_registry/generate_csv.py` around lines 1 - 6, Add
the standard NVIDIA Apache-2.0 copyright header (including the year of latest
meaningful modification) to the top of generate_csv.py above the module
docstring and imports; ensure the header text matches other TensorRT-LLM source
files (Apache 2.0 license boilerplate and NVIDIA copyright line) so the file
complies with project guidelines.

tensorrt_llm/_torch/auto_deploy/models/custom/__init__.py-1-2 (1)

1-2: ⚠️ Potential issue | 🟡 Minor

Restore the standard NVIDIA Apache-2.0 header.

This modified Python source file now starts directly with imports. Please add the required license header back at the top. As per coding guidelines, "Add NVIDIA copyright header on ALL new files, and update year on modified files."

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tensorrt_llm/_torch/auto_deploy/models/custom/__init__.py` around lines 1 -
2, This file (__init__.py) is missing the required NVIDIA Apache-2.0 license
header; add the standard NVIDIA Apache-2.0 copyright header block at the very
top of tensorrt_llm/_torch/auto_deploy/models/custom/__init__.py (above the
existing imports of CohereForCausalLM and DeciLMForCausalLM), updating the
copyright year as needed to reflect modification.

.codex/skills/ad-model-onboard/SKILL.md-156-160 (1)

156-160: ⚠️ Potential issue | 🟡 Minor

Fix the world_size_N.yaml size buckets.

Lines 158-160 leave 15B-20B models uncovered and make 80B ambiguous between the 4-GPU and 8-GPU buckets. Please make the ranges exhaustive and non-overlapping.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In @.codex/skills/ad-model-onboard/SKILL.md around lines 156 - 160, Update the
world_size_N.yaml size buckets to be exhaustive and non-overlapping: define
world_size_1.yaml for models <2B, world_size_2.yaml for models from 2B up to and
including 15B, world_size_4.yaml for models >15B and <80B, and world_size_8.yaml
for models >=80B; modify the explanatory bullets in SKILL.md (the section
describing the yaml_extra selection and the "Pick world_size_N.yaml based on
model size" text) accordingly so the ranges are explicit and unambiguous while
keeping the instruction to always include dashboard_default.yaml first and to
append any model-specific YAML after the world_size entry.

.codex/skills/ad-model-onboard/SKILL.md-245-247 (1)

245-247: ⚠️ Potential issue | 🟡 Minor

Add a language to this fenced command block.

This currently trips markdownlint MD040. Use bash like the surrounding examples.

Suggested fix

-```
+```bash
 python examples/auto_deploy/build_and_run_ad.py --model <MODEL-ID> --use-registry

</details>

<details>
<summary>🤖 Prompt for AI Agents</summary>

Verify each finding against the current code and only fix it if needed.

In @.codex/skills/ad-model-onboard/SKILL.md around lines 245 - 247, The fenced
code block containing the command python examples/auto_deploy/build_and_run_ad.py --model <MODEL-ID> --use-registry is
missing a language tag and triggers markdownlint MD040; update that
triple-backtick fence to include "bash" (i.e., change tobash) so the
block matches surrounding examples and satisfies the linter.

</details>

</blockquote></details>
<details>
<summary>tensorrt_llm/_torch/auto_deploy/models/custom/modeling_deepseek_v2.py-1-1 (1)</summary><blockquote>

`1-1`: _⚠️ Potential issue_ | _🟡 Minor_

**Update copyright year to 2026.**

The copyright year shows 2025, but since this is a new file created in March 2026, it should be updated to 2026 to match other new files in this PR.


<details>
<summary>📝 Proposed fix</summary>

```diff
-# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-FileCopyrightText: Copyright (c) 2022-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
```
</details>

<details>
<summary>🤖 Prompt for AI Agents</summary>

```
Verify each finding against the current code and only fix it if needed.

In `@tensorrt_llm/_torch/auto_deploy/models/custom/modeling_deepseek_v2.py` at
line 1, Update the SPDX copyright header year from 2025 to 2026 at the top of
the file (the file-level header comment in modeling_deepseek_v2.py) so the
header reads 2026, matching the rest of the PR.
```

</details>

</blockquote></details>
<details>
<summary>.claude/skills/ad-model-onboard/SKILL.md-17-19 (1)</summary><blockquote>

`17-19`: _⚠️ Potential issue_ | _🟡 Minor_

**Explicitly sum the `nvidia-smi` output or reword this step.**

`nvidia-smi --query-gpu=memory.total --format=csv,noheader,nounits` returns one value per GPU, not a single total. As written, Step 0 can be read as comparing the model size against one line of output instead of the aggregate VRAM budget it mentions.

<details>
<summary>🤖 Prompt for AI Agents</summary>

```
Verify each finding against the current code and only fix it if needed.

In @.claude/skills/ad-model-onboard/SKILL.md around lines 17 - 19, The step
currently suggests running `nvidia-smi --query-gpu=memory.total
--format=csv,noheader,nounits` but doesn't clarify that it returns per-GPU
values; update the SKILL.md step to either instruct users to sum those per-GPU
values (e.g., "sum the returned lines to get total VRAM across all GPUs") or
reword the step to explicitly say "run the command and sum the per-GPU outputs
to compute total system VRAM" so the model size is compared to the aggregate
VRAM budget rather than a single GPU value.
```

</details>

</blockquote></details>
<details>
<summary>.claude/skills/ad-model-onboard/SKILL.md-158-160 (1)</summary><blockquote>

`158-160`: _⚠️ Potential issue_ | _🟡 Minor_

**The size buckets skip 16B-19B models.**

The current guidance covers `<2B`, `2-15B`, `20-80B`, and `80B+`, so 16B/17B/18B/19B family members have no prescribed `world_size_N.yaml`. Please close that gap so the onboarding rule stays deterministic.

<details>
<summary>🤖 Prompt for AI Agents</summary>

```
Verify each finding against the current code and only fix it if needed.

In @.claude/skills/ad-model-onboard/SKILL.md around lines 158 - 160, Update the
size-bucket guidance in SKILL.md so 16–19B models are covered: keep the rule to
"Always include dashboard_default.yaml first", then change the world_size
mapping to explicitly include "world_size_1.yaml for <2B, world_size_2.yaml for
2–15B, world_size_4.yaml for 16–80B, world_size_8.yaml for 80B+" (or
alternatively add a separate "16–19B -> world_size_4.yaml" bullet if you prefer
minimal change); ensure any mention of "world_size" or "world_size_N.yaml" in
the same section is updated to reflect the new bucket so onboarding guidance is
deterministic.
```

</details>

</blockquote></details>

</blockquote></details>

<details>
<summary>🧹 Nitpick comments (13)</summary><blockquote>

<details>
<summary>tensorrt_llm/_torch/auto_deploy/models/custom/modeling_gpt_oss.py (1)</summary><blockquote>

`467-493`: **Consider adding error handling for dequantization failures.**

The `convert_moe_packed_tensors` call could fail (e.g., invalid tensor shapes, dtype mismatches) and would raise an unhandled exception during model loading. Consider wrapping the conversion in a try-except to provide a clearer error message.


<details>
<summary>♻️ Proposed improvement for error handling</summary>

```diff
     for base, blocks_key, scales_key in keys_to_process:
-        blocks = state_dict.pop(blocks_key).cpu()
-        scales = state_dict.pop(scales_key).cpu()
-        dequantized = convert_moe_packed_tensors(blocks, scales, dtype=torch.bfloat16)
-        state_dict[base] = dequantized
+        try:
+            blocks = state_dict.pop(blocks_key).cpu()
+            scales = state_dict.pop(scales_key).cpu()
+            dequantized = convert_moe_packed_tensors(blocks, scales, dtype=torch.bfloat16)
+            state_dict[base] = dequantized
+        except Exception as e:
+            raise RuntimeError(
+                f"Failed to dequantize MXFP4 weights for {base}: {e}"
+            ) from e
```
</details>

<details>
<summary>🤖 Prompt for AI Agents</summary>

```
Verify each finding against the current code and only fix it if needed.

In `@tensorrt_llm/_torch/auto_deploy/models/custom/modeling_gpt_oss.py` around
lines 467 - 493, Wrap the convert_moe_packed_tensors call inside
_mxfp4_dequant_load_hook in a try/except so dequantization failures produce a
clear contextual error instead of crashing silently; catch exceptions around
converting each (blocks_key, scales_key) pair, on error restore any popped
entries back into state_dict (blocks_key and scales_key) and either raise a new
RuntimeError with details including base/blocks_key/scales_key and the original
exception message or log the error before re-raising, so callers can identify
which expert tensor failed and why.
```

</details>

</blockquote></details>
<details>
<summary>tensorrt_llm/_torch/auto_deploy/models/custom/modeling_hunyuan_moe.py (1)</summary><blockquote>

`179-188`: **Minor: Unused unpacked variables.**

The variables `T` and `D` from the shape unpacking are never used. Consider prefixing with underscore for clarity.


<details>
<summary>✏️ Suggested fix</summary>

```diff
     def forward(self, hidden_states: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]:
         """Return (topk_indices, topk_weights), shapes [T, K] each."""
-        T, D = hidden_states.shape
+        _T, _D = hidden_states.shape
         # Cast both input and weight to float32 for gate computation
         logits = F.linear(hidden_states.float(), self.wg.weight.float())
```

Or simply remove the unpacking if not needed for documentation:
```diff
-        T, D = hidden_states.shape
```
</details>

<details>
<summary>🤖 Prompt for AI Agents</summary>

```
Verify each finding against the current code and only fix it if needed.

In `@tensorrt_llm/_torch/auto_deploy/models/custom/modeling_hunyuan_moe.py` around
lines 179 - 188, In the forward method of class/model handling routing (function
forward), remove the unused local variables T and D or rename them to _T and _D
(or simply use a single underscore _) when unpacking hidden_states.shape to
avoid unused-variable warnings; update the line "T, D = hidden_states.shape"
accordingly and ensure no other code relies on T or D.
```

</details>

</blockquote></details>
<details>
<summary>tensorrt_llm/_torch/auto_deploy/models/custom/modeling_llama4.py (2)</summary><blockquote>

`179-185`: **Verify the input shape assumption for expert computation.**

The reshape on line 180 assumes `hidden_states` has shape `[num_experts * tokens, hidden_size]` and reshapes to `[num_experts, tokens, hidden_size]`. This shape is set by the caller (`Llama4MoE.forward`), but the assumption isn't immediately obvious and could break if the MoE layer is used differently.

Consider adding a shape assertion or comment documenting the expected input contract.


<details>
<summary>📝 Suggested documentation</summary>

```diff
     def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
+        # Input shape: [num_experts * num_tokens, hidden_size]
+        # Reshaped to: [num_experts, num_tokens, hidden_size] for parallel expert computation
         hidden_states = hidden_states.view(self.gate_up_proj.shape[0], -1, self.hidden_size)
```
</details>

<details>
<summary>🤖 Prompt for AI Agents</summary>

```
Verify each finding against the current code and only fix it if needed.

In `@tensorrt_llm/_torch/auto_deploy/models/custom/modeling_llama4.py` around
lines 179 - 185, The forward method in modeling_llama4.py assumes hidden_states
is shaped as [num_experts * tokens, hidden_size] and reshapes it using
self.gate_up_proj.shape[0] to [num_experts, tokens, hidden_size]; add an
explicit check and/or clear inline comment documenting this contract so misuse
is caught early. In the forward function (method name: forward) reference
hidden_states, self.gate_up_proj, self.down_proj and the caller
Llama4MoE.forward in the comment and add an assertion that hidden_states.dim()
== 2 and hidden_states.size(1) == self.hidden_size and that
hidden_states.size(0) is divisible by self.gate_up_proj.shape[0] (or otherwise
matches expected num_experts * tokens) before the view operation.
```

</details>

---

`614-623`: **Consider guarding the HF vision model import for robustness.**

The import of `Llama4VisionModel` from transformers at line 617 only occurs inside `__init__`, unlike other Llama4 imports at module level. While the project pins `transformers==4.57.1` (which includes Llama4VisionModel support since 4.51.0), adding error handling would make the code more defensive against unexpected version mismatches or environments where the model is unavailable.

<details>
<summary>🛡️ Proposed improvement</summary>

```diff
     def __init__(self, config: Llama4Config, **kwargs):
         super().__init__(config)
         # Import HF's vision model for weight loading compatibility
-        from transformers.models.llama4.modeling_llama4 import (
-            Llama4VisionModel as HFLlama4VisionModel,
-        )
+        try:
+            from transformers.models.llama4.modeling_llama4 import (
+                Llama4VisionModel as HFLlama4VisionModel,
+            )
+        except ImportError as e:
+            raise ImportError(
+                "Llama4VisionModel requires transformers >= 4.51.0 with Llama 4 support"
+            ) from e
 
         self.vision_model = HFLlama4VisionModel(config.vision_config)
```
</details>

<details>
<summary>🤖 Prompt for AI Agents</summary>

```
Verify each finding against the current code and only fix it if needed.

In `@tensorrt_llm/_torch/auto_deploy/models/custom/modeling_llama4.py` around
lines 614 - 623, The inline import of Llama4VisionModel inside __init__ (used to
set self.vision_model) is not guarded and can raise an unclear ImportError on
mismatched transformers versions; wrap the import of Llama4VisionModel in a
try/except ImportError inside __init__ (or perform a module-level guarded
import) and on failure raise a clear RuntimeError that explains the missing
transformer class and required version, or provide a sensible fallback (e.g.,
set self.vision_model = None) so downstream code can handle absence gracefully;
reference the __init__, Llama4VisionModel, and self.vision_model symbols when
applying the change.
```

</details>

</blockquote></details>
<details>
<summary>tensorrt_llm/_torch/auto_deploy/models/custom/modeling_phi4_visionr.py (1)</summary><blockquote>

`1-14`: **Copyright header format differs from other files.**

This file uses `Copyright (c) 2026` while other new files in this PR use `Copyright (c) 2022-2026`. Consider aligning with the established pattern for consistency.



<details>
<summary>📝 Suggested header alignment</summary>

```diff
-# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+# SPDX-FileCopyrightText: Copyright (c) 2022-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
```
</details>

<details>
<summary>🤖 Prompt for AI Agents</summary>

```
Verify each finding against the current code and only fix it if needed.

In `@tensorrt_llm/_torch/auto_deploy/models/custom/modeling_phi4_visionr.py`
around lines 1 - 14, Update the copyright header in modeling_phi4_visionr.py to
match the project's standard range by changing "Copyright (c) 2026" to
"Copyright (c) 2022-2026" at the top of the file; keep the rest of the Apache
License boilerplate unchanged so the header format is consistent with other new
files in the PR.
```

</details>

</blockquote></details>
<details>
<summary>tensorrt_llm/_torch/auto_deploy/custom_ops/fla/torch_backend_gated_delta.py (1)</summary><blockquote>

`196-203`: **Clarify variable naming: `num_k_heads` is derived from Q's shape, not K's.**

The variable `num_k_heads` at line 197 is assigned from `q.shape[2]` (Q's head count), not K's shape. This naming is misleading. The condition at line 199 (`num_v_heads > num_k_heads`) is intentional and documented in the FLA gated delta rule: Q/K have fewer heads than V. However, the variable name obscures this intent. Consider renaming to `num_qk_heads` to clarify that Q and K share the same head count, which differs from V.

<details>
<summary>📝 Suggested clarification</summary>

```diff
     # Handle GQA: expand Q/K heads to match V heads if needed
-    num_k_heads = q.shape[2]
+    num_qk_heads = q.shape[2]  # Q and K share the same head count
     num_v_heads = v.shape[2]
-    if num_v_heads > num_k_heads:
-        n_rep = num_v_heads // num_k_heads
+    if num_v_heads > num_qk_heads:
+        n_rep = num_v_heads // num_qk_heads
         q = q.repeat_interleave(n_rep, dim=2)
         k = k.repeat_interleave(n_rep, dim=2)
```
</details>

<details>
<summary>🤖 Prompt for AI Agents</summary>

```
Verify each finding against the current code and only fix it if needed.

In `@tensorrt_llm/_torch/auto_deploy/custom_ops/fla/torch_backend_gated_delta.py`
around lines 196 - 203, Rename the misleading variable num_k_heads (assigned
from q.shape[2]) to num_qk_heads to make it clear Q and K share the same head
count; update the condition (num_v_heads > num_qk_heads) and the subsequent
repeat_interleave calls on q and k to use num_qk_heads, and adjust the comment
above the block to mention "Q/K heads" explicitly; locate this change in
torch_backend_gated_delta.py around the block that references q, k, v and
num_v_heads.
```

</details>

</blockquote></details>
<details>
<summary>examples/auto_deploy/model_registry/configs/hunyuan_mt_7b.yaml (1)</summary><blockquote>

`1-4`: **Consider adding at least an empty YAML mapping or a placeholder key.**

This file contains only comments with no YAML keys. While YAML parsers typically handle comment-only files gracefully (returning `null` or empty), some registry implementations may expect at least an empty document. If `dashboard_default` truly provides all necessary values, consider adding a brief placeholder like `{}` or `# (uses dashboard_default)` to make the intent explicit in the parsed structure.



<details>
<summary>💡 Optional: Add empty mapping for clarity</summary>

```diff
 # Configuration for Tencent Hunyuan-MT-7B (Dense)
 # Standard dense GQA model with QK norm and Dynamic NTK-Alpha RoPE.
 # Uses a custom AutoDeploy model implementation (HunYuanDenseForCausalLM)
 # registered locally; no additional overrides needed beyond dashboard_default.
+
+# Empty config - inherits all settings from dashboard_default
+{}
```
</details>

<details>
<summary>🤖 Prompt for AI Agents</summary>

```
Verify each finding against the current code and only fix it if needed.

In `@examples/auto_deploy/model_registry/configs/hunyuan_mt_7b.yaml` around lines
1 - 4, Add an explicit empty YAML document or placeholder to make the intent
explicit and ensure parsers/registries don't return null; update
hunyuan_mt_7b.yaml to include either an empty mapping ("{}") or a comment
placeholder like "# uses dashboard_default" so the file parses to a non-null
document while still relying on dashboard_default for values referenced in the
config.
```

</details>

</blockquote></details>
<details>
<summary>.codex/agents/ad_debug_agent.toml (1)</summary><blockquote>

`14-18`: **CLI flag inconsistency with the Markdown agent documentation.**

This TOML uses `--args.yaml-extra examples/auto_deploy/model_registry/configs/<CONFIG_YAML_FILE>` (line 17), but `.claude/agents/ad-debug-agent.md` uses `--use-registry` (line 60). The PR's registry-driven workflow suggests `--use-registry` is the preferred approach since it auto-resolves configs from `models.yaml`.

Consider aligning this TOML with the registry-based workflow:


<details>
<summary>♻️ Proposed fix to align with registry workflow</summary>

```diff
 Typical run command:
-AD_DUMP_GRAPHS_DIR=<AD_DUMP_GRAPHS_DIR> python examples/auto_deploy/build_and_run_ad.py \
+CUDA_VISIBLE_DEVICES=<SELECTED_GPUS> AD_DUMP_GRAPHS_DIR=<AD_DUMP_GRAPHS_DIR> \
+  python examples/auto_deploy/build_and_run_ad.py \
   --model <MODEL_HF_ID> \
-  --args.yaml-extra examples/auto_deploy/model_registry/configs/<CONFIG_YAML_FILE> \
+  --use-registry \
   2>&1 | tee <LOG_FILE>
```
</details>

<details>
<summary>🤖 Prompt for AI Agents</summary>

```
Verify each finding against the current code and only fix it if needed.

In @.codex/agents/ad_debug_agent.toml around lines 14 - 18, Update the CLI
example in the TOML to use the registry-driven flag to match the Markdown docs:
replace the usage of the --args.yaml-extra flag in the Typical run command with
the --use-registry flag so the example follows the registry workflow (i.e., run
examples/auto_deploy/build_and_run_ad.py with --use-registry to auto-resolve
configs from models.yaml), ensuring the flags --args.yaml-extra and
--use-registry are not both shown as the canonical example.
```

</details>

</blockquote></details>
<details>
<summary>examples/auto_deploy/model_registry/generate_csv.py (2)</summary><blockquote>

`21-23`: **Specify explicit encoding for file operations.**

For cross-platform compatibility, explicitly specify `encoding="utf-8"` when opening text files.


<details>
<summary>📝 Add explicit encoding</summary>

```diff
-    with open(MODELS_YAML) as f:
+    with open(MODELS_YAML, encoding="utf-8") as f:
         data = yaml.safe_load(f)
```

And for the output file:

```diff
-    with open(OUTPUT_CSV, "w", newline="") as f:
+    with open(OUTPUT_CSV, "w", newline="", encoding="utf-8") as f:
```
</details>

<details>
<summary>🤖 Prompt for AI Agents</summary>

```
Verify each finding against the current code and only fix it if needed.

In `@examples/auto_deploy/model_registry/generate_csv.py` around lines 21 - 23,
The file operations in main() currently open MODELS_YAML (and the CSV output)
without an explicit encoding; update the open(...) calls to include
encoding="utf-8" when reading MODELS_YAML in main() and when writing the CSV
output file (wherever the write/open for the CSV is performed) to ensure
cross-platform UTF-8 handling; locate the open calls in main() and the
CSV-writing function/section and add encoding="utf-8".
```

</details>

---

`9-9`: **Unused constant `REPO_ROOT`.**

`REPO_ROOT` is defined but never referenced in the script. Consider removing it if it's not needed.


<details>
<summary>🧹 Remove unused constant</summary>

```diff
-# Paths relative to TensorRT-LLM repo root
-REPO_ROOT = Path(__file__).resolve().parents[3]
 MODELS_YAML = Path(__file__).resolve().parent / "models.yaml"
```
</details>

<details>
<summary>🤖 Prompt for AI Agents</summary>

```
Verify each finding against the current code and only fix it if needed.

In `@examples/auto_deploy/model_registry/generate_csv.py` at line 9, The constant
REPO_ROOT is declared but not used in generate_csv.py; remove the unused symbol
REPO_ROOT (the Path(__file__).resolve().parents[3] assignment) to eliminate dead
code, or if it was intended to be used for resolving paths, replace its usage by
referencing REPO_ROOT in the relevant functions (e.g., in CSV generation or file
reads) instead of hardcoding paths; target the REPO_ROOT declaration to either
delete or wire into path resolution logic.
```

</details>

</blockquote></details>
<details>
<summary>.codex/agents/ad_run_agent.toml (1)</summary><blockquote>

`18-22`: **Inconsistent with `.claude/agents/ad-run-agent.md`: consider using `--use-registry`.**

This workflow uses `--args.yaml-extra` directly, while the corresponding `.claude/agents/ad-run-agent.md` uses `--use-registry` for automatic config resolution from the registry. Consider aligning on `--use-registry` for consistency, unless there's a specific reason to require explicit YAML paths here.


<details>
<summary>📝 Update to use --use-registry</summary>

```diff
 1. Run:
-AD_DUMP_GRAPHS_DIR=<AD_DUMP_GRAPHS_DIR> python examples/auto_deploy/build_and_run_ad.py \
+CUDA_VISIBLE_DEVICES=<SELECTED_GPUS> AD_DUMP_GRAPHS_DIR=<AD_DUMP_GRAPHS_DIR> python examples/auto_deploy/build_and_run_ad.py \
   --model <MODEL_HF_ID> \
-  --args.yaml-extra examples/auto_deploy/model_registry/configs/<CONFIG_YAML_FILE> \
+  --use-registry \
   2>&1 | tee <LOG_FILE>
```
</details>

<details>
<summary>🤖 Prompt for AI Agents</summary>

```
Verify each finding against the current code and only fix it if needed.

In @.codex/agents/ad_run_agent.toml around lines 18 - 22, The command in
ad_run_agent.toml uses the explicit flag --args.yaml-extra to pass a config file
to examples/auto_deploy/build_and_run_ad.py, but the documentation
(.claude/agents/ad-run-agent.md) expects the --use-registry flow; update the Run
snippet to use --use-registry instead of --args.yaml-extra (or add an
explanatory comment if explicit YAML is required) so the CLI invocation in this
file matches the automatic registry-based config resolution described in the
docs; reference build_and_run_ad.py and the flags --args.yaml-extra and
--use-registry when making the change.
```

</details>

</blockquote></details>
<details>
<summary>.claude/agents/ad-run-agent.md (1)</summary><blockquote>

`86-88`: **Add language identifier to fenced code block.**

The code block showing the log path format should have a language identifier (or use empty identifier for plain text).


<details>
<summary>📝 Add language identifier</summary>

```diff
-```
+```text
 $PWD/ad-test-workspace/ad_run_logs/<MODEL_SHORT_NAME>_<YYYYMMDD_HHMMSS>_<STATUS>.log
 ```
```
</details>

<details>
<summary>🤖 Prompt for AI Agents</summary>

Verify each finding against the current code and only fix it if needed.

In @.claude/agents/ad-run-agent.md around lines 86 - 88, The fenced code block
that displays the log path
"$PWD/ad-test-workspace/ad_run_logs/<MODEL_SHORT_NAME><YYYYMMDD_HHMMSS>.log"
lacks a language identifier; update that Markdown block to include a language
tag such as "text" (i.e., change totext) so the snippet is treated as
plain text and renders consistently.

</details>

</blockquote></details>
<details>
<summary>tensorrt_llm/_torch/auto_deploy/custom_ops/fla/fla_gated_delta.py (1)</summary><blockquote>

`160-166`: **Add validation for GQA head count divisibility.**

The GQA expansion assumes `num_v_heads` is evenly divisible by `num_k_heads`. If this invariant is violated, the resulting tensor shape will be incorrect without any clear error message.


<details>
<summary>🛡️ Add divisibility assertion</summary>

```diff
     # Handle GQA: expand Q/K heads to match V heads if needed
     num_k_heads = q.shape[2]
     num_v_heads = v.shape[2]
     if num_v_heads > num_k_heads:
+        assert num_v_heads % num_k_heads == 0, (
+            f"GQA requires num_v_heads ({num_v_heads}) to be divisible by "
+            f"num_k_heads ({num_k_heads})"
+        )
         n_rep = num_v_heads // num_k_heads
         q = q.repeat_interleave(n_rep, dim=2)
         k = k.repeat_interleave(n_rep, dim=2)
```
</details>

<details>
<summary>🤖 Prompt for AI Agents</summary>

```
Verify each finding against the current code and only fix it if needed.

In `@tensorrt_llm/_torch/auto_deploy/custom_ops/fla/fla_gated_delta.py` around
lines 160 - 166, The code assumes num_v_heads is divisible by num_k_heads before
using n_rep = num_v_heads // num_k_heads and repeat_interleave on q and k; add a
validation check right after computing num_k_heads and num_v_heads that raises a
clear ValueError (or AssertionError) when num_v_heads % num_k_heads != 0
(include the actual values in the message), and only perform n_rep = num_v_heads
// num_k_heads and q = q.repeat_interleave(n_rep, dim=2); k =
k.repeat_interleave(n_rep, dim=2) when the divisibility check passes to avoid
silent shape errors.
```

</details>

</blockquote></details>

</blockquote></details>

---

<details>
<summary>ℹ️ Review info</summary>

<details>
<summary>⚙️ Run configuration</summary>

**Configuration used**: Path: .coderabbit.yaml

**Review profile**: CHILL

**Plan**: Pro

**Run ID**: `3859f7f0-d841-4b1e-abe2-67386567f9fd`

</details>

<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between 2fb5e805335d2cd0a9e580797a9d39b85cfbfc58 and 89a5aeaafc7b1a34fa0e38f9975537f1bb79d41b.

</details>

<details>
<summary>📒 Files selected for processing (138)</summary>

* `.claude/agents/ad-debug-agent.md`
* `.claude/agents/ad-onboard-reviewer.md`
* `.claude/agents/ad-run-agent.md`
* `.claude/skills/ad-model-onboard/SKILL.md`
* `.codex/AGENTS.md`
* `.codex/agents/ad_debug_agent.toml`
* `.codex/agents/ad_onboard_reviewer.toml`
* `.codex/agents/ad_run_agent.toml`
* `.codex/agents/onboard_update_reviewer.toml`
* `.codex/config.toml`
* `.codex/skills/ad-model-onboard/SKILL.md`
* `AGENTS.md`
* `docs/source/models/supported-models.md`
* `examples/auto_deploy/build_and_run_ad.py`
* `examples/auto_deploy/model_registry/configs/dashboard_default.yaml`
* `examples/auto_deploy/model_registry/configs/deepseek_v2_ep.yaml`
* `examples/auto_deploy/model_registry/configs/exaone.yaml`
* `examples/auto_deploy/model_registry/configs/glm4_moe.yaml`
* `examples/auto_deploy/model_registry/configs/glm_5.yaml`
* `examples/auto_deploy/model_registry/configs/gpt_oss.yaml`
* `examples/auto_deploy/model_registry/configs/granite_4.0_h_small.yaml`
* `examples/auto_deploy/model_registry/configs/granite_4.0_micro.yaml`
* `examples/auto_deploy/model_registry/configs/granite_4.0_tiny_preview.yaml`
* `examples/auto_deploy/model_registry/configs/hunyuan_mt_7b.yaml`
* `examples/auto_deploy/model_registry/configs/minimax_m2.yaml`
* `examples/auto_deploy/model_registry/configs/nano_v3.yaml`
* `examples/auto_deploy/model_registry/configs/nemotron_flash.yaml`
* `examples/auto_deploy/model_registry/configs/nemotron_super_49b.yaml`
* `examples/auto_deploy/model_registry/configs/openelm.yaml`
* `examples/auto_deploy/model_registry/configs/phi4-multimodal-instruct.yaml`
* `examples/auto_deploy/model_registry/configs/phi4-reasoning-vision-15b.yaml`
* `examples/auto_deploy/model_registry/configs/qwen3.5_dense.yaml`
* `examples/auto_deploy/model_registry/configs/super_v3.yaml`
* `examples/auto_deploy/model_registry/generate_csv.py`
* `examples/auto_deploy/model_registry/models.yaml`
* `tensorrt_llm/_torch/auto_deploy/config/default.yaml`
* `tensorrt_llm/_torch/auto_deploy/custom_ops/README.md`
* `tensorrt_llm/_torch/auto_deploy/custom_ops/attention/trtllm_attention.py`
* `tensorrt_llm/_torch/auto_deploy/custom_ops/attention_interface.py`
* `tensorrt_llm/_torch/auto_deploy/custom_ops/fla/fla_backend_gated_delta.py`
* `tensorrt_llm/_torch/auto_deploy/custom_ops/fla/fla_gated_delta.py`
* `tensorrt_llm/_torch/auto_deploy/custom_ops/fla/torch_backend_gated_delta.py`
* `tensorrt_llm/_torch/auto_deploy/custom_ops/linear/torch_router.py`
* `tensorrt_llm/_torch/auto_deploy/custom_ops/mamba/torch_backend_mamba.py`
* `tensorrt_llm/_torch/auto_deploy/custom_ops/mamba/torch_mamba.py`
* `tensorrt_llm/_torch/auto_deploy/llm.py`
* `tensorrt_llm/_torch/auto_deploy/llm_args.py`
* `tensorrt_llm/_torch/auto_deploy/models/custom/__init__.py`
* `tensorrt_llm/_torch/auto_deploy/models/custom/modeling_cohere.py`
* `tensorrt_llm/_torch/auto_deploy/models/custom/modeling_decilm.py`
* `tensorrt_llm/_torch/auto_deploy/models/custom/modeling_deepseek_v2.py`
* `tensorrt_llm/_torch/auto_deploy/models/custom/modeling_exaone.py`
* `tensorrt_llm/_torch/auto_deploy/models/custom/modeling_gemma.py`
* `tensorrt_llm/_torch/auto_deploy/models/custom/modeling_gemma2.py`
* `tensorrt_llm/_torch/auto_deploy/models/custom/modeling_gemma3.py`
* `tensorrt_llm/_torch/auto_deploy/models/custom/modeling_glm4_moe.py`
* `tensorrt_llm/_torch/auto_deploy/models/custom/modeling_glm_moe_dsa.py`
* `tensorrt_llm/_torch/auto_deploy/models/custom/modeling_gpt_oss.py`
* `tensorrt_llm/_torch/auto_deploy/models/custom/modeling_granite.py`
* `tensorrt_llm/_torch/auto_deploy/models/custom/modeling_granite_moe_hybrid.py`
* `tensorrt_llm/_torch/auto_deploy/models/custom/modeling_hunyuan_dense.py`
* `tensorrt_llm/_torch/auto_deploy/models/custom/modeling_hunyuan_moe.py`
* `tensorrt_llm/_torch/auto_deploy/models/custom/modeling_internlm3.py`
* `tensorrt_llm/_torch/auto_deploy/models/custom/modeling_llama3.py`
* `tensorrt_llm/_torch/auto_deploy/models/custom/modeling_llama4.py`
* `tensorrt_llm/_torch/auto_deploy/models/custom/modeling_minimax_m2.py`
* `tensorrt_llm/_torch/auto_deploy/models/custom/modeling_mistral.py`
* `tensorrt_llm/_torch/auto_deploy/models/custom/modeling_mistral3.py`
* `tensorrt_llm/_torch/auto_deploy/models/custom/modeling_olmo3.py`
* `tensorrt_llm/_torch/auto_deploy/models/custom/modeling_openelm.py`
* `tensorrt_llm/_torch/auto_deploy/models/custom/modeling_phi4.py`
* `tensorrt_llm/_torch/auto_deploy/models/custom/modeling_phi4_visionr.py`
* `tensorrt_llm/_torch/auto_deploy/models/custom/modeling_phi4flash.py`
* `tensorrt_llm/_torch/auto_deploy/models/custom/modeling_phi4mm.py`
* `tensorrt_llm/_torch/auto_deploy/models/custom/modeling_qwen2.py`
* `tensorrt_llm/_torch/auto_deploy/models/custom/modeling_qwen3.py`
* `tensorrt_llm/_torch/auto_deploy/models/custom/modeling_qwen3_5.py`
* `tensorrt_llm/_torch/auto_deploy/models/custom/modeling_qwen3_moe.py`
* `tensorrt_llm/_torch/auto_deploy/models/custom/modeling_qwen3_next.py`
* `tensorrt_llm/_torch/auto_deploy/models/custom/modeling_seed_oss.py`
* `tensorrt_llm/_torch/auto_deploy/models/custom/modeling_skywork_r1v2.py`
* `tensorrt_llm/_torch/auto_deploy/models/custom/modeling_smollm3.py`
* `tensorrt_llm/_torch/auto_deploy/models/custom/modeling_starcoder2.py`
* `tensorrt_llm/_torch/auto_deploy/models/patches/gptoss-mxfp4.py`
* `tensorrt_llm/_torch/auto_deploy/models/patches/llama4.py`
* `tensorrt_llm/_torch/auto_deploy/models/patches/minimax_m2.py`
* `tensorrt_llm/_torch/auto_deploy/models/patches/mistral3.py`
* `tensorrt_llm/_torch/auto_deploy/models/patches/qwen3.py`
* `tensorrt_llm/_torch/auto_deploy/models/patches/qwen3_next.py`
* `tensorrt_llm/_torch/auto_deploy/shim/interface.py`
* `tensorrt_llm/_torch/auto_deploy/transform/library/fuse_quant.py`
* `tensorrt_llm/_torch/auto_deploy/transform/library/rms_norm.py`
* `tensorrt_llm/_torch/auto_deploy/utils/benchmark.py`
* `tests/unittest/_torch/auto_deploy/unit/singlegpu/models/test_gemma3_modeling.py`
* `tests/unittest/auto_deploy/singlegpu/custom_ops/test_resource_handlers.py`
* `tests/unittest/auto_deploy/singlegpu/models/test_cohere_modeling.py`
* `tests/unittest/auto_deploy/singlegpu/models/test_decilm_modeling.py`
* `tests/unittest/auto_deploy/singlegpu/models/test_deepseek_v2_modeling.py`
* `tests/unittest/auto_deploy/singlegpu/models/test_exaone_modeling.py`
* `tests/unittest/auto_deploy/singlegpu/models/test_gemma2_modeling.py`
* `tests/unittest/auto_deploy/singlegpu/models/test_gemma_modeling.py`
* `tests/unittest/auto_deploy/singlegpu/models/test_glm4_moe_modeling.py`
* `tests/unittest/auto_deploy/singlegpu/models/test_glm_moe_dsa_modeling.py`
* `tests/unittest/auto_deploy/singlegpu/models/test_gpt_oss_modeling.py`
* `tests/unittest/auto_deploy/singlegpu/models/test_granite_modeling.py`
* `tests/unittest/auto_deploy/singlegpu/models/test_granite_moe_hybrid_modeling.py`
* `tests/unittest/auto_deploy/singlegpu/models/test_hunyuan_dense_modeling.py`
* `tests/unittest/auto_deploy/singlegpu/models/test_hunyuan_moe_modeling.py`
* `tests/unittest/auto_deploy/singlegpu/models/test_internlm3_modeling.py`
* `tests/unittest/auto_deploy/singlegpu/models/test_llama3_modeling.py`
* `tests/unittest/auto_deploy/singlegpu/models/test_llama4_modeling.py`
* `tests/unittest/auto_deploy/singlegpu/models/test_minimax_m2_modeling.py`
* `tests/unittest/auto_deploy/singlegpu/models/test_mistral3_modeling.py`
* `tests/unittest/auto_deploy/singlegpu/models/test_mistral3_patches.py`
* `tests/unittest/auto_deploy/singlegpu/models/test_mistral_modeling.py`
* `tests/unittest/auto_deploy/singlegpu/models/test_olmo3_modeling.py`
* `tests/unittest/auto_deploy/singlegpu/models/test_openelm_modeling.py`
* `tests/unittest/auto_deploy/singlegpu/models/test_phi4_modeling.py`
* `tests/unittest/auto_deploy/singlegpu/models/test_phi4_visionr_modeling.py`
* `tests/unittest/auto_deploy/singlegpu/models/test_phi4flash_modeling.py`
* `tests/unittest/auto_deploy/singlegpu/models/test_phi4mm_modeling.py`
* `tests/unittest/auto_deploy/singlegpu/models/test_qwen2_modeling.py`
* `tests/unittest/auto_deploy/singlegpu/models/test_qwen3_5_modeling.py`
* `tests/unittest/auto_deploy/singlegpu/models/test_qwen3_modeling.py`
* `tests/unittest/auto_deploy/singlegpu/models/test_qwen3_moe_modeling.py`
* `tests/unittest/auto_deploy/singlegpu/models/test_qwen3_next_gdn_patches.py`
* `tests/unittest/auto_deploy/singlegpu/models/test_qwen3_next_modeling.py`
* `tests/unittest/auto_deploy/singlegpu/models/test_qwen3_next_patches.py`
* `tests/unittest/auto_deploy/singlegpu/models/test_seed_oss_modeling.py`
* `tests/unittest/auto_deploy/singlegpu/models/test_skywork_r1v2_modeling.py`
* `tests/unittest/auto_deploy/singlegpu/models/test_smollm3_modeling.py`
* `tests/unittest/auto_deploy/singlegpu/models/test_starcoder2_modeling.py`
* `tests/unittest/auto_deploy/singlegpu/shim/test_cached_sequence_interface.py`
* `tests/unittest/auto_deploy/singlegpu/shim/test_engine.py`
* `tests/unittest/auto_deploy/singlegpu/transformations/library/test_gated_delta_rule_cache.py`
* `tests/unittest/auto_deploy/singlegpu/transformations/library/test_kv_cache.py`
* `tests/unittest/auto_deploy/singlegpu/transformations/library/test_torch_gated_delta_rule_cache.py`
* `tests/unittest/auto_deploy/singlegpu/utils/test_benchmark_mlp.py`

</details>

<details>
<summary>💤 Files with no reviewable changes (1)</summary>

* examples/auto_deploy/model_registry/configs/dashboard_default.yaml

</details>

</details>

<!-- This is an auto-generated comment by CodeRabbit for review status -->

[coderabbitai]

⚠️ Potential issue | 🟠 Major

Do not synthesize position_ids in the export path.

AutoDeploy prefills need explicit positions. Defaulting to 0..S-1 hides a missing input and gives wrong RoPE for non-zero-offset or packed sequences.

Suggested fix

-        batch_size, seq_len = inputs_embeds.shape[:2]
-        if position_ids is None:
-            position_ids = (
-                torch.arange(seq_len, device=inputs_embeds.device)
-                .unsqueeze(0)
-                .expand(batch_size, -1)
-            )
+        if position_ids is None:
+            raise ValueError("position_ids must be provided for AD export")

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tensorrt_llm/_torch/auto_deploy/models/custom/modeling_gemma3.py` around
lines 414 - 420, The current code synthesizes position_ids from inputs_embeds
(using torch.arange and expand), which must not happen in the export/AutoDeploy
path; update the block that creates position_ids so that if position_ids is None
and we are in an export/tracing/scripting context (torch.jit.is_tracing() or
torch.jit.is_scripting()), we raise a clear ValueError stating position_ids must
be provided for export instead of synthesizing them; keep the existing fallback
synthesis only for normal runtime (non-export) paths so references to
inputs_embeds, position_ids, and the surrounding forward (or the method
containing this code) remain correct.

[coderabbitai]

⚠️ Potential issue | 🟠 Major

This wrapper is registered as image-text, but its forward path is still text-only.

Gemma3ForConditionalGeneration.forward never uses vision_tower or multi_modal_projector, and the current kwarg filtering drops multimodal inputs before the call into Gemma3ForCausalLM. Because the class is also registered in AutoModelForImageTextToTextFactory, multimodal runs can silently ignore the image input instead of failing fast. That is especially risky because the registry still marks google/gemma-3-27b-it with multimodal.yaml.

Suggested fix

     def forward(
         self,
         input_ids: Optional[torch.LongTensor] = None,
         position_ids: Optional[torch.LongTensor] = None,
         inputs_embeds: Optional[torch.FloatTensor] = None,
         **kwargs,
     ) -> Gemma3CausalLMOutput:
+        if "pixel_values" in kwargs:
+            raise NotImplementedError(
+                "Gemma3ForConditionalGeneration export path is text-only; image inputs are not supported."
+            )
         language_model_kwargs = {}

If image-text support is not intended yet, please also drop the AutoModelForImageTextToTextFactory registration until the vision path is wired.

Also applies to: 567-569

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tensorrt_llm/_torch/auto_deploy/models/custom/modeling_gemma3.py` around
lines 532 - 560, The forward wrapper (Gemma3ForConditionalGeneration.forward) is
registered as image-text but currently only forwards text args to
self.language_model and filters out multimodal kwargs; either wire the vision
path or fail fast: detect multimodal inputs (e.g., pixel_values / vision_tower /
multi_modal_projector related kwargs) before the existing kwargs filtering in
forward and if self.vision_tower and self.multi_modal_projector are available,
run the vision_tower and multi_modal_projector to produce inputs_embeds and
include them in language_model_kwargs, otherwise raise a clear ValueError
stating multimodal inputs are unsupported (or remove the
AutoModelForImageTextToTextFactory registration) so image inputs are not
silently dropped. Ensure checks reference the symbols language_model.forward,
vision_tower, multi_modal_projector, and language_model_kwargs so the fix is
easy to locate.

[coderabbitai]

⚠️ Potential issue | 🟠 Major

Add the standard SPDX/Apache header to this new file.

This new TensorRT-LLM source file is missing the required Apache 2.0 header block, and the year should reflect the 2026 addition.

Suggested fix

-# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.

As per coding guidelines, "All TensorRT-LLM source files should contain an NVIDIA copyright header with the year of the latest meaningful modification. The header should be an Apache 2.0 license block as specified" and "Add NVIDIA copyright header on ALL new files, and update year on modified files".

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
	# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
	# SPDX-License-Identifier: Apache-2.0
	#
	# Licensed under the Apache License, Version 2.0 (the "License");
	# you may not use this file except in compliance with the License.
	# You may obtain a copy of the License at
	#
	# http://www.apache.org/licenses/LICENSE-2.0
	#
	# Unless required by applicable law or agreed to in writing, software
	# distributed under the License is distributed on an "AS IS" BASIS,
	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	# See the License for the specific language governing permissions and
	# limitations under the License.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tensorrt_llm/_torch/auto_deploy/models/custom/modeling_glm_moe_dsa.py` at
line 1, This file
tensorrt_llm/_torch/auto_deploy/models/custom/modeling_glm_moe_dsa.py is missing
the required NVIDIA Apache-2.0 header; add the standard SPDX/Apache header block
at the top of the file with the NVIDIA copyright line and the year 2026, using
the exact license text and SPDX identifier (Apache-2.0) required by project
guidelines so the header precedes the module code in modeling_glm_moe_dsa.py.

[coderabbitai]

⚠️ Potential issue | 🟠 Major

Replace these asserts with explicit input validation.

Optimized Python removes assert, so position_ids becomes optional at runtime and the model fails later with a less actionable error.

Suggested fix

-        assert position_ids is not None, "position_ids is required"
+        if position_ids is None:
+            raise ValueError("position_ids is required")

-        assert position_ids is not None, "position_ids is required"
+        if position_ids is None:
+            raise ValueError("position_ids is required")

Also applies to: 659-659

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tensorrt_llm/_torch/auto_deploy/models/custom/modeling_glm_moe_dsa.py` at
line 597, Replace the runtime-removed asserts for input validation: instead of
using "assert position_ids is not None, 'position_ids is required'" (and the
similar check at the other occurrence), add explicit validation that checks if
position_ids is None and raise a clear exception (e.g., ValueError or TypeError)
with the message "position_ids is required". Update the checks in
modeling_glm_moe_dsa.py where position_ids is validated so the function/method
fails fast with a clear error at runtime rather than relying on assert.

[coderabbitai]

⚠️ Potential issue | 🟠 Major

Remove the direct torch.ops.trtllm dependency from the router.

Line 155 calls torch.ops.trtllm.noaux_tc_op, but the onboarding contract updated in this PR only allows torch.ops.auto_deploy.torch_* or plain PyTorch in model code. This will also break the required CPU unit-test path. If helpful, I can sketch the pure-PyTorch sigmoid/bias/group-topk rewrite.

🧰 Tools

🪛 Ruff (0.15.6)

[warning] 148-148: Unpacked variable bsz is never used

Prefix it with an underscore or any other dummy variable pattern

(RUF059)

---

[warning] 148-148: Unpacked variable seq_len is never used

Prefix it with an underscore or any other dummy variable pattern

(RUF059)

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tensorrt_llm/_torch/auto_deploy/models/custom/modeling_glm4_moe.py` around
lines 147 - 164, The forward method currently calls torch.ops.trtllm.noaux_tc_op
(in modeling_glm4_moe.forward) which violates the new onboarding contract and
breaks CPU unit tests; replace that call with either a pure-PyTorch
implementation of the fused operations (apply bias, sigmoid, perform group-wise
top-k selection, normalize and scale using the existing tensors router_logits,
self.e_score_correction_bias, self.n_group, self.topk_group, self.top_k,
self.routed_scaling_factor) or call the approved operator namespace
(torch.ops.auto_deploy.*) if an equivalent op exists; ensure the replacement
reproduces the original outputs topk_indices and topk_weights and runs on CPU.

[coderabbitai]

⚠️ Potential issue | 🟠 Major

Keep query length separate from max position.

Line 280 overwrites seq_len with position_ids.max() + 1, but Line 308 still uses seq_len for the final reshape. If a caller passes offset or padded position_ids, that reshape size no longer matches the actual query length.

🐛 Proposed fix

     def forward(self, hidden_states: torch.Tensor, position_ids: torch.LongTensor) -> torch.Tensor:
-        batch_size, seq_len, _ = hidden_states.shape
+        batch_size, query_len, _ = hidden_states.shape
         qkv = self.qkv_proj(hidden_states)
         query_pos = self.num_heads * self.head_dim
         kv_pos = self.num_key_value_heads * self.head_dim
         query_states = qkv[..., :query_pos]
         key_states = qkv[..., query_pos : query_pos + kv_pos]
         value_states = qkv[..., query_pos + kv_pos :]
 
-        query_states = query_states.view(batch_size, seq_len, self.num_heads, self.head_dim)
-        key_states = key_states.view(batch_size, seq_len, self.num_key_value_heads, self.head_dim)
+        query_states = query_states.view(batch_size, query_len, self.num_heads, self.head_dim)
+        key_states = key_states.view(batch_size, query_len, self.num_key_value_heads, self.head_dim)
         value_states = value_states.view(
-            batch_size, seq_len, self.num_key_value_heads, self.head_dim
+            batch_size, query_len, self.num_key_value_heads, self.head_dim
         )
 
         position_embeddings = self.rotary_emb(value_states)
         if self.rotary_emb.rope_scaling is None:
             cos, sin = position_embeddings
         else:
             short_cos, short_sin, long_cos, long_sin = position_embeddings
             if position_ids.is_meta:
                 cos, sin = short_cos, short_sin
             else:
-                seq_len = int(position_ids.max().item()) + 1
-                if seq_len <= self.rotary_emb.original_max_position_embeddings:
+                max_position = int(position_ids.max().item()) + 1
+                if max_position <= self.rotary_emb.original_max_position_embeddings:
                     cos, sin = short_cos, short_sin
                 else:
                     cos, sin = long_cos, long_sin
         cos = cos[position_ids]
         sin = sin[position_ids]
@@
         attn_output = torch.ops.auto_deploy.torch_attention(
             query_states,
             key_states,
             value_states,
             is_causal=True,
             scale=self.scaling,
             layout="bsnd",
         )
-        attn_output = attn_output.reshape(batch_size, seq_len, self.num_heads * self.head_dim)
+        attn_output = attn_output.reshape(batch_size, query_len, self.num_heads * self.head_dim)
         return self.o_proj(attn_output)

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tensorrt_llm/_torch/auto_deploy/models/custom/modeling_phi4mm.py` around
lines 257 - 309, The method forward overwrites the original query length stored
in seq_len when computing seq_len = int(position_ids.max().item()) + 1, which
breaks the final reshape of attn_output; preserve the original batch_size and
seq_len for tensor shapes by using a separate variable (e.g. pos_seq_len or
rope_seq_len) when inspecting position_ids and choosing cos/sin in the rotary
embedding logic (inside forward, around the rotary_emb/position_ids handling)
instead of reassigning seq_len so attn_output.reshape(batch_size, seq_len, ...)
still uses the true query length.

[coderabbitai]

⚠️ Potential issue | 🟠 Major

Require position_ids instead of synthesizing them.

Line 726 makes position_ids optional again and generates a single [1, S] row for the whole batch. That changes the exported interface and can hide upstream caller bugs that should have been caught at the boundary.

🛡️ Proposed change

-        if position_ids is None:
-            seq_len = inputs_embeds.shape[1]
-            position_ids = torch.arange(seq_len, device=inputs_embeds.device).unsqueeze(0)
+        if position_ids is None:
+            raise ValueError("position_ids must be provided for AD export")

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tensorrt_llm/_torch/auto_deploy/models/custom/modeling_phi4mm.py` around
lines 726 - 728, The code currently synthesizes position_ids when None (using
inputs_embeds.shape[1]) which hides caller errors and changes the exported
interface; instead, make position_ids mandatory by removing the fallback
generation and adding an explicit check that raises a clear error if
position_ids is None (e.g., in the same function/method where position_ids is
used, reference the position_ids parameter and inputs_embeds to validate), so
callers must supply a [batch_size, seq_len] tensor; ensure any
docstring/signature for the forward/inference method (the function using
position_ids in modeling_phi4mm.py) reflects this required parameter and the
raised error message points to the missing position_ids input.

[coderabbitai]

⚠️ Potential issue | 🟠 Major

Add the standard 2026 Apache header to this new file.

Line 1 still says 2025, and the file is missing the Apache 2.0 header block used in the other new source files under this directory. That leaves the new file out of compliance with the repo's required header format.

As per coding guidelines **/*.{cpp,h,cu,cuh,hpp,py}: All TensorRT-LLM source files should contain an NVIDIA copyright header with the year of the latest meaningful modification. The header should be an Apache 2.0 license block as specified.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tensorrt_llm/_torch/auto_deploy/models/custom/modeling_qwen3_5.py` around
lines 1 - 22, Update the file header in modeling_qwen3_5.py to include the
standard 2026 NVIDIA Apache-2.0 license block used across the project (replace
the current "2025" year and the missing header); insert the full Apache 2.0
header/license comment at the top of the file matching other TensorRT-LLM source
files and ensure the copyright line references 2026 and NVIDIA CORPORATION,
leaving the rest of the module (e.g., the Qwen3.5 docstring and functions)
unchanged.

[coderabbitai]

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🌐 Web query:

Does Python remove assertstatements when running with the-Oflag, and should unsupported code paths raiseNotImplementedErrororRuntimeError instead?

💡 Result:

Yes. Running Python with -O (“optimize”) removes assert statements from the generated bytecode (and also removes code guarded by if __debug__: by setting __debug__ to False).[1] So assert must not be relied on for required validation or control flow.

For “unsupported” code paths, pick the exception based on why it’s unsupported:

• Use NotImplementedError when the object is intended to support the operation, but the implementation is intentionally missing (e.g., abstract base-class method, stub during development).[2]
• Use RuntimeError for an unexpected internal situation that “doesn’t fall into any other category”.[2] For “should be unreachable” paths where you still want the failure in optimized runs, raise an exception explicitly (e.g., raise RuntimeError("unreachable")), rather than assert False.

Sources: [1] Python “Setup and Usage” (-O removes asserts / __debug__-guarded code). [2] Python docs: built-in exceptions (NotImplementedError, RuntimeError).

---

🏁 Script executed:

# Check if file exists and get line count
wc -l tensorrt_llm/_torch/auto_deploy/models/custom/modeling_qwen3_5.py

Repository: NVIDIA/TensorRT-LLM

Length of output: 133

---

🏁 Script executed:

# Read the specific lines mentioned in the review (1391-1405)
sed -n '1391,1405p' tensorrt_llm/_torch/auto_deploy/models/custom/modeling_qwen3_5.py

Repository: NVIDIA/TensorRT-LLM

Length of output: 823

---

🏁 Script executed:

# Search for other instances of assert False in this file
grep -n "assert False" tensorrt_llm/_torch/auto_deploy/models/custom/modeling_qwen3_5.py

Repository: NVIDIA/TensorRT-LLM

Length of output: 95

---

🏁 Script executed:

# Get broader context around line 1395
sed -n '1370,1420p' tensorrt_llm/_torch/auto_deploy/models/custom/modeling_qwen3_5.py

Repository: NVIDIA/TensorRT-LLM

Length of output: 2967

---

🏁 Script executed:

# Check CODING_GUIDELINES.md for exception handling guidance
find . -name "CODING_GUIDELINES.md" -type f

Repository: NVIDIA/TensorRT-LLM

Length of output: 85

---

🏁 Script executed:

# Check CODING_GUIDELINES.md for exception handling recommendations
cat CODING_GUIDELINES.md | grep -A 20 -B 2 "exception\|Exception\|assert\|Assert"

Repository: NVIDIA/TensorRT-LLM

Length of output: 3787

---

Replace assert False with raise NotImplementedError() for unsupported vision path.

Line 1395 uses assert False, which Python removes entirely when run with the -O flag (optimize mode). In optimized runs, this guard disappears and execution continues into a path explicitly marked unsupported. Use raise NotImplementedError() instead to ensure the guard persists across all execution modes.

Proposed change

         if has_vision:
             # Multimodal: compute mRoPE positions with spatial (T, H, W) layout.
             # NOTE: This path needs the mRoPE position delta cache transform to work
             # correctly with the AD runtime during decode steps.
-            assert False, (
+            raise NotImplementedError(
                 "Vision path not yet supported in AD runtime. Requires mRoPE position "
                 "delta cache transform. See: "
                 "https://github.com/nv-auto-deploy/TensorRT-LLM/pull/189#discussion_r2915084063"
             )
-            position_ids, _ = self.get_rope_index(
-                input_ids,
-                image_grid_thw=image_grid_thw,
-                video_grid_thw=video_grid_thw,
-                attention_mask=attention_mask,
-            )

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	if has_vision:
	# Multimodal: compute mRoPE positions with spatial (T, H, W) layout.
	# NOTE: This path needs the mRoPE position delta cache transform to work
	# correctly with the AD runtime during decode steps.
	assert False, (
	"Vision path not yet supported in AD runtime. Requires mRoPE position "
	"delta cache transform. See: "
	"https://github.com/nv-auto-deploy/TensorRT-LLM/pull/189#discussion_r2915084063"
	)
	position_ids, _ = self.get_rope_index(
	input_ids,
	image_grid_thw=image_grid_thw,
	video_grid_thw=video_grid_thw,
	attention_mask=attention_mask,
	)
	if has_vision:
	# Multimodal: compute mRoPE positions with spatial (T, H, W) layout.
	# NOTE: This path needs the mRoPE position delta cache transform to work
	# correctly with the AD runtime during decode steps.
	raise NotImplementedError(
	"Vision path not yet supported in AD runtime. Requires mRoPE position "
	"delta cache transform. See: "
	"https://github.com/nv-auto-deploy/TensorRT-LLM/pull/189#discussion_r2915084063"
	)

🧰 Tools

🪛 Ruff (0.15.6)

[warning] 1395-1395: Do not assert False (python -O removes these calls), raise AssertionError()

Replace assert False

(B011)

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tensorrt_llm/_torch/auto_deploy/models/custom/modeling_qwen3_5.py` around
lines 1391 - 1405, The guard in the vision branch (when has_vision is true in
the block that calls self.get_rope_index) uses assert False which is stripped
under Python -O; replace that assert with raise NotImplementedError(...) so the
unsupported vision path always raises at runtime; update the exception text to
keep the same explanatory message referencing the mRoPE position delta cache
transform and the PR link to aid debugging (keep reference to
self.get_rope_index and the vision/multimodal comment).

[coderabbitai]

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🌐 Web query:

Does transformers.AutoConfig.registeraccept anexist_ok keyword in the Transformers version used by this repo, and if not, what backward-compatible registration pattern should be used?

💡 Result:

No—if this repo is on an older Transformers where AutoConfig.register is defined as register(model_type, config) (e.g., Transformers v4.12.0), it does not accept an exist_ok keyword. In v4.12.0 the method signature has no exist_ok parameter. [1] By contrast, in newer Transformers (e.g., v4.42.4) the documented signature is register(model_type, config, exist_ok=False). [2]

A backward-compatible pattern is to feature-detect (or catch TypeError) and fall back:

from transformers import AutoConfig
import inspect

def register_autoconfig(model_type: str, config_cls):
    sig = inspect.signature(AutoConfig.register)
    if "exist_ok" in sig.parameters:
        AutoConfig.register(model_type, config_cls, exist_ok=True)
        return

    # Old Transformers: no exist_ok
    try:
        AutoConfig.register(model_type, config_cls)
    except ValueError:
        # already registered; ignore only if it's the same class
        from transformers.models.auto.configuration_auto import CONFIG_MAPPING
        if CONFIG_MAPPING[model_type] is not config_cls:
            raise

Sources:
[1] Transformers v4.12.0 AutoConfig.register(model_type, config) implementation (no exist_ok). [1]
[2] Transformers v4.42.4 docs showing AutoConfig.register(model_type, config, exist_ok=False). [2]

---

🏁 Script executed:

# Check the target file at the specified lines
cat -n tensorrt_llm/_torch/auto_deploy/models/custom/modeling_qwen3_5.py | sed -n '1475,1485p'

Repository: NVIDIA/TensorRT-LLM

Length of output: 651

---

🏁 Script executed:

# Find other custom-model files to verify the existing pattern
find tensorrt_llm -name "modeling_*.py" -path "*/custom/*" -type f

Repository: NVIDIA/TensorRT-LLM

Length of output: 2891

---

🏁 Script executed:

# Check Transformers version requirement
fd -t f -e txt -e toml -e py -e cfg . | xargs grep -l "transformers" | head -10

Repository: NVIDIA/TensorRT-LLM

Length of output: 594

---

🏁 Script executed:

# Check a few other custom-model files for the error-handling pattern
head -50 tensorrt_llm/_torch/auto_deploy/models/custom/modeling_cohere.py | tail -30

Repository: NVIDIA/TensorRT-LLM

Length of output: 1332

---

🏁 Script executed:

# Look at the end of a custom model file to see registration pattern
tail -20 tensorrt_llm/_torch/auto_deploy/models/custom/modeling_cohere.py

Repository: NVIDIA/TensorRT-LLM

Length of output: 810

---

🏁 Script executed:

# Check another one
tail -20 tensorrt_llm/_torch/auto_deploy/models/custom/modeling_deepseek.py

Repository: NVIDIA/TensorRT-LLM

Length of output: 763

---

🏁 Script executed:

# Find Transformers version requirement
find . -maxdepth 2 -name "pyproject.toml" -o -name "setup.py" -o -name "requirements*.txt" | xargs grep -i transformers 2>/dev/null | head -10

Repository: NVIDIA/TensorRT-LLM

Length of output: 223

---

🏁 Script executed:

# Search for all uses of AutoConfig.register in custom modeling files
rg "AutoConfig\.register" tensorrt_llm/_torch/auto_deploy/models/custom/ -B 2 -A 2

Repository: NVIDIA/TensorRT-LLM

Length of output: 8444

---

Guard AutoConfig.register(..., exist_ok=True) calls with try-except for backward compatibility with older Transformers.

Lines 1480-1481 will fail on Transformers versions that do not support the exist_ok parameter. Several other custom-model files already use the try-except fallback pattern (e.g., modeling_kimi_k2.py, modeling_exaone.py, modeling_phi4flash.py). Apply the same pattern here:

Proposed change

-AutoConfig.register("qwen3_5", Qwen3_5Config, exist_ok=True)
-AutoConfig.register("qwen3_5_text", Qwen3_5TextConfig, exist_ok=True)
+for model_type, config_cls in (
+    ("qwen3_5", Qwen3_5Config),
+    ("qwen3_5_text", Qwen3_5TextConfig),
+):
+    try:
+        AutoConfig.register(model_type, config_cls, exist_ok=True)
+    except TypeError:
+        try:
+            AutoConfig.register(model_type, config_cls)
+        except ValueError:
+            pass

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	AutoConfig.register("qwen3_5", Qwen3_5Config, exist_ok=True)
	AutoConfig.register("qwen3_5_text", Qwen3_5TextConfig, exist_ok=True)
	for model_type, config_cls in (
	("qwen3_5", Qwen3_5Config),
	("qwen3_5_text", Qwen3_5TextConfig),
	):
	try:
	AutoConfig.register(model_type, config_cls, exist_ok=True)
	except TypeError:
	try:
	AutoConfig.register(model_type, config_cls)
	except ValueError:
	pass

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tensorrt_llm/_torch/auto_deploy/models/custom/modeling_qwen3_5.py` around
lines 1480 - 1481, The AutoConfig.register calls for Qwen3_5Config and
Qwen3_5TextConfig use the exist_ok=True parameter which breaks on older
Transformers; wrap both registrations in a try-except that first calls
AutoConfig.register("qwen3_5", Qwen3_5Config, exist_ok=True) and
AutoConfig.register("qwen3_5_text", Qwen3_5TextConfig, exist_ok=True) inside the
try, and in the except TypeError (or generic Exception) fall back to calling
AutoConfig.register("qwen3_5", Qwen3_5Config) and
AutoConfig.register("qwen3_5_text", Qwen3_5TextConfig) so the code remains
compatible with versions that lack the exist_ok parameter.

[galagam]

Please rebase, the nano config already exists in this location, just update the relevant config (sharding_source: ['manual'] was removed here)

[galagam]

Same as nanov3, the file exists, please rebase and add any relevant changes