[#12332][feat] AutoDeploy: SuperV3 MTP Support

[govind-ramnarayan]

Resolves: #12332

Summary by CodeRabbit

Release Notes

• New Features

  • Added support for MTP (Multi-Token Prediction) speculative decoding alongside existing Eagle3 one-model support.
  • Introduced intermediate state caching for speculative decoding to optimize memory and performance.

• Refactor

  • Generalized Eagle model architecture to support multiple model types (Llama, NemotronH) for improved flexibility.
  • Restructured attention and custom operations to support optional speculative decoding configuration.
  • Enhanced resource handler system with specialized handlers for speculative decoding scenarios.

• Tests

  • Added integration tests for MTP-based speculative decoding.
  • Added unit tests for speculative resource handlers and model weight loading.
  • Expanded smoke tests for speculative decoding workflows.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

[tensorrt-cicd]

PR_Github #40746 [ run ] completed with state ABORTED. Commit: f6bd237

Link to invocation

[venkywonka]

lgtm from llmapi side

[tensorrt-cicd]

PR_Github #40755 [ run ] completed with state SUCCESS. Commit: 12ae4e3
/LLM/main/L0_MergeRequest_PR pipeline #31773 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[govind-ramnarayan]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #40840 [ run ] triggered by Bot. Commit: 12ae4e3 Link to invocation

[tensorrt-cicd]

PR_Github #40840 [ run ] completed with state SUCCESS. Commit: 12ae4e3
/LLM/main/L0_MergeRequest_PR pipeline #31849 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[govind-ramnarayan]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #40991 [ run ] triggered by Bot. Commit: 7b587ca Link to invocation

[tensorrt-cicd]

PR_Github #40991 [ run ] completed with state FAILURE. Commit: 7b587ca
/LLM/main/L0_MergeRequest_PR pipeline #31972 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[govind-ramnarayan]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #41101 [ run ] triggered by Bot. Commit: 042fbbf Link to invocation

[tensorrt-cicd]

PR_Github #41101 [ run ] completed with state SUCCESS. Commit: 042fbbf
/LLM/main/L0_MergeRequest_PR pipeline #32075 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[govind-ramnarayan]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #41234 [ run ] triggered by Bot. Commit: 042fbbf Link to invocation

[govind-ramnarayan]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #41266 [ run ] triggered by Bot. Commit: 2c429f6 Link to invocation

[tensorrt-cicd]

PR_Github #41266 [ run ] completed with state SUCCESS. Commit: 2c429f6
/LLM/main/L0_MergeRequest_PR pipeline #32224 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[govind-ramnarayan]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #41366 [ run ] triggered by Bot. Commit: 2c429f6 Link to invocation

[tensorrt-cicd]

PR_Github #41366 [ run ] completed with state SUCCESS. Commit: 2c429f6
/LLM/main/L0_MergeRequest_PR pipeline #32309 completed with status: 'SUCCESS'

CI Report

Link to invocation