Skip to content

[TRTLLM-12440][feat] Add GMS-only weight sharing support#13926

Draft
chienchunhung wants to merge 1 commit intoNVIDIA:mainfrom
chienchunhung:trtllm-12440-gms-only
Draft

[TRTLLM-12440][feat] Add GMS-only weight sharing support#13926
chienchunhung wants to merge 1 commit intoNVIDIA:mainfrom
chienchunhung:trtllm-12440-gms-only

Conversation

@chienchunhung
Copy link
Copy Markdown
Collaborator

@chienchunhung chienchunhung commented May 9, 2026

@coderabbitai summary

Summary

Adds LoadFormat.GMS so multiple TRT-LLM instances on the same node can zero-copy share model weights via the GPU Memory Service (GMS) pool. The first instance loads weights as the writer (RW); subsequent peers materialize them read-only (RO) without disk I/O or per-instance copies.

Scope

In scope:

  • LoadFormat.GMS enum value and nested GmsConfig (socket_path, mode, tag) on TorchLlmArgs.
  • New GMSBackend adapter under tensorrt_llm/_torch/memory/, lazily imported only when GMS is selected.
  • ModelLoader GMS branch with explicit RW / RO / unexpected-state handling, plus a guard that refuses to commit an unpopulated model to the pool.
  • Cleanup hook released via PyTorchModelEngine.__del__.

Adjacent (sub-PR, ~15 lines): remove the redundant MXCheckpointLoader.p2p_succeeded property; is_weights_preloaded() (the abstract hook from BaseCheckpointLoader) is now the single accessor. Tests updated accordingly.

Out of scope:

Test Coverage

New unit test files (mock-based, CPU CI):

  • tests/unittest/llmapi/test_gms_args.pyGmsConfig validation and LoadFormat.GMS Pydantic surface.
  • tests/unittest/_torch/memory/test_gms_backend.pyGMSBackend lifecycle and helpers.
  • tests/unittest/_torch/pyexecutor/test_model_loader_gms.pyModelLoader GMS RW / RO / failure / edge-case branches.

Updated:

  • tests/unittest/_torch/models/checkpoints/mx/test_mx_checkpoint_loader.py — six call sites switched to is_weights_preloaded() after removing the redundant p2p_succeeded property.

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
@chienchunhung
Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd
Copy link
Copy Markdown
Collaborator

PR_Github #47456 [ run ] triggered by Bot. Commit: 03ddff2 Link to invocation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants