Skip to content

fix: resolve PermissionError during local mode cleanup of root-owned Docker files#5629

Draft
mollyheamazon wants to merge 5 commits intoaws:masterfrom
mollyheamazon:fix/local-mode
Draft

fix: resolve PermissionError during local mode cleanup of root-owned Docker files#5629
mollyheamazon wants to merge 5 commits intoaws:masterfrom
mollyheamazon:fix/local-mode

Conversation

@mollyheamazon
Copy link
Contributor

Issue

Local container training fails with PermissionError during post-training cleanup.
Docker containers write files as root, and shutil.rmtree cannot remove them when
running as a non-root user (e.g., sagemaker-user in Studio).

Root Cause

The original _rmtree attempted os.chmod on PermissionError, which also fails
on root-owned files when running as non-root.

Fix

On PermissionError, use docker run chmod -R 777 (as root inside the container)
to fix permissions, then retry shutil.rmtree. In SageMaker Studio, --network sagemaker is added since Studio restricts Docker to that network. The already-pulled
training image is reused to avoid pulling additional images.

Changes

  • sagemaker-train/.../local/local_container.py — updated _rmtree, pass image
    and is_studio from train() cleanup calls
  • sagemaker-core/.../local_core/local_container.py — same fix, kept in sync
  • Added unit tests for _rmtree covering normal, fallback, Studio, and error cases

Testing

  • Reproduced and verified fix on SageMaker Studio (Docker-enabled JupyterLab space)
  • Unit tests pass for both sagemaker-train (5 new) and sagemaker-core (31 existing)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@mollyheamazon mollyheamazon deployed to auto-approve March 14, 2026 00:10 — with GitHub Actions Active
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant