fix(gcp): remove unsupported --include flags from gcloud storage rsync#799
Open
arniechops wants to merge 13 commits intomainfrom
Open
fix(gcp): remove unsupported --include flags from gcloud storage rsync#799arniechops wants to merge 13 commits intomainfrom
arniechops wants to merge 13 commits intomainfrom
Conversation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
gcloud storage rsync only supports --exclude (Python regex), not --include. The --include flags caused rsync to fail on GCP model weight downloads. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
model-engine/model_engine_server/domain/use_cases/llm_model_endpoint_use_cases.py
Outdated
Show resolved
Hide resolved
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
gcloud storage rsynconly supports--exclude(Python regex), not--include--includeflags caused model weight downloads to fail on GCP with:unrecognized arguments: --include=*.model--includeflags and kept only--exclude="optimizer.*"since all other files in the checkpoint bucket are neededTest plan
🤖 Generated with Claude Code
Greptile Summary
This PR fixes a GCP-specific bug where
gcloud storage rsyncwas being called with--includeflags that it doesn't support, causing model weight downloads to fail on GCP withunrecognized arguments: --include=*.model. The fix removes the unsupported--includeflags and switches to--exclude-only patterns using Python regex syntax (as required bygcloud storage rsync). Thetrust_remote_codesecurity boundary — which was lost in an earlier draft — has been restored here by adding--exclude=".*\.py$"whentrust_remote_code=False, keeping behavior consistent with the Azure/S3 code paths.Key changes:
--includeflags from thegcloud storage rsyncinvocation inload_model_weights_sub_commands_gcsoptimizer*) to a Python regex (optimizer.*)--exclude=".*\.py$"whentrust_remote_code=Falseto prevent downloading Python code from untrusted checkpoints--excludeis usedConfidence Score: 5/5
Safe to merge — fixes a clear crash-level bug with correct regex syntax and preserves the trust_remote_code security boundary
The change is minimal and targeted: it removes flags that were definitively unsupported by gcloud storage rsync, uses the correct --exclude-only approach with valid Python regex patterns, and restores the .py-file exclusion guard for untrusted checkpoints. No P1 or P0 issues found; the previous concern about trust_remote_code has been fully addressed in this revision.
No files require special attention
Important Files Changed
Flowchart
%%{init: {'theme': 'neutral'}}%% flowchart TD A[load_model_weights_sub_commands_gcs] --> B[Install gcloud CLI] B --> C[Build exclude filters] C --> D["Always: --exclude='optimizer.*'"] D --> E{trust_remote_code?} E -- False --> F["Add: --exclude='.*\\.py$'"] E -- True --> G[No additional excludes] F --> H[gcloud storage rsync -r with excludes] G --> H H --> I[Sync all other files from GCS checkpoint]Reviews (2): Last reviewed commit: "fix(gcp): exclude .py files from gcloud ..." | Re-trigger Greptile