Add support for packed modules to keep deployments <250mb #38

sean-r-williams · 2025-06-24T03:31:45Z

Issue #, if available:
Related to #37.

Description of changes:
This PR adds support for customers to deploy compressed archives of modules, then unpack them at runtime. This provides a compelling option for customers with large sets of dependent modules, who would otherwise be constrained to 250MB minus the size of the PowerShell runtime (currently ~172 MB runtime size --> 78 MB dependency size).

Two forms of compressed dependencies are supported:

Module archives: A ZIP file, named modules.zip. The contents of this archive are unpacked to a subdirectory of /tmp as-is. This format assumes someone's zipping a folder in $env:PSModulePath (or somewhere they saved several modules into) and trades layer support/reproducability for better compression ratios.
Module packages: A collection of .nupkg files, in a subdirectory named module-nupkgs. These packages are "saved" (installed) to a subdirectory of /tmp via PSResourceGet. This should broadly align with the ideas behind Lambda's layer architecture, and allows for direct package usage (as a binary, without unpacking/etc.) off a given NuGet feed.

Both forms of dependencies:

Can be used interchangeably or simultaneously.
Pull from /opt/ (for layers) or $env:LAMBDA_TASK_ROOT (for function packages).
- The latter should take precedence.
Extract to separate subdirectories of /tmp, to minimize chances of module packaging formats conflicting with each other.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

sean-r-williams · 2025-06-24T03:32:18Z

N.B.: Marking this as draft while I run some tests locally - definitely open to feedback/review while I'm doing this, however.

…ll-runtime into sean-r-williams/37_compressed-modules

sean-r-williams · 2025-07-18T18:42:09Z

Ran some tests in our account with modules.zip (replicating analogous logic into the function itself instead of the runtime). Unpacking to /tmp works as expected - I didn't have an opportunity to test .nupkg in-situ but I was able to ad-hoc test Save-PSResource with a filesystem directory containing a bunch of NuGet packages.

@austoonz @afroz429, let me know what your thoughts are on this.

austoonz

I do like this concept! Super useful.

Comments are somewhat focused primarily from a performance aspect of reducing touch points to the file system.

austoonz · 2025-07-18T19:43:01Z

powershell-runtime/source/bootstrap

+# Unpack compressed modules, if present
+
+# Combined
+If (Test-RuntimePackedModule -Combined) {


For performance reasons (reading from disk is relatively slow), it would be great to merge these into a single call, that looks for both .zip and .nupkg files, and passes the files found into appropriate import functions.

As it stands, this will traverse the file system twice (in each of the Test-RuntimePackedModule calls, and then if something is found, again in the relative Import-* functions).

Perhaps the Test-RuntimePackedModule could return the specific paths to the files found, and the Import-* functions could accept a string of path values as a parameters reducing the disk reads.

austoonz · 2025-07-18T19:47:01Z

powershell-runtime/source/modules/Private/Test-RuntimePackedModule.ps1

+    $BaseDirectories = @(
+        "/opt",
+        $Env:LAMBDA_TASK_ROOT
+    )


If this was call in bootstrap after the Set-PSModulePath call, perhaps this could use the paths configured in $env:PSModulePath for the evaluation?

Because this is called before Set-PSModulePath (and dirs Set-PSModulePath looks for are only populated because this cmdlet says so), I don't think relying on contents of $env:PSModulePath would work as expected.

I could potentially put these paths in a separate $Script: scope variable, but given the very specific use case I'm not sure they'd be of any use.

powershell-runtime/source/modules/Private/Import-ModuleArchive.ps1

austoonz · 2025-07-18T19:55:36Z

powershell-runtime/source/modules/Private/Import-ModuleArchive.ps1

+    )
+
+    If ($SearchPaths | ? { Test-Path $_ }) {
+        $UnpackDirectory = '/tmp/powershell-custom-runtime-unpacked-modules/combined/'


This might be best to define in pwsh-runtime.psm1 as a script (/module) scoped variable. That will mean it can also be used to configure $env:PSModulePath in Set-PSModulePath, allowing the unpacked modules to be imported as normal.

db51cda hoisted these variables into the script scope and d2fb0e9 updated the Import-Module* cmdlets to use the script-scope dirs. Let me know what you think.

powershell-runtime/source/modules/Private/Import-ModuleArchive.ps1

austoonz · 2025-07-18T20:08:41Z

powershell-runtime/source/modules/Private/Import-ModulePackage.ps1

+            $PackageDirectory = Split-Path $_.Value -Parent
+            if ($env:POWERSHELL_RUNTIME_VERBOSE -eq 'TRUE') { Write-Host "[RUNTIME-Import-ModulePackage]Importing module packages from $PackageDirectory" }
+            $RepositoryName = "Lambda-Local-$($_.Key)"
+            if ($env:POWERSHELL_RUNTIME_VERBOSE -eq 'TRUE') { Write-Host "[RUNTIME-Import-ModulePackage]Registering local package repository $RepositoryName" }
+            Register-PSResourceRepository -Name $RepositoryName -Uri $PackageDirectory -Trusted -Priority 1
+            if ($env:POWERSHELL_RUNTIME_VERBOSE -eq 'TRUE') { Write-Host "[RUNTIME-Import-ModulePackage]Enumerating packages in $PackageDirectory (PSResource repository $RepositoryName)" }
+            Find-PSResource -Name * -Repository $RepositoryName | ForEach-Object -Parallel {
+                if ($env:POWERSHELL_RUNTIME_VERBOSE -eq 'TRUE') { Write-Host "[RUNTIME-Import-ModulePackage]Saving package $($_.Name) version $($_.Version) (PSResource repository $($using:RepositoryName))" }
+                $_ | Save-PSResource -SkipDependencyCheck -Path $using:PackageDirectory -Quiet -AcceptLicense -Confirm:$false
+            }
+            if ($env:POWERSHELL_RUNTIME_VERBOSE -eq 'TRUE') { Write-Host "[RUNTIME-Import-ModulePackage]Registering local package repository $RepositoryName" }
+            Unregister-PSResourceRepository -Name $RepositoryName -Confirm:$false
+        }


Do these need to be invoked for every item from $SearchPaths.GetEnumerator()?

Instead, perhaps each file should be stored appropriately, and then a single repository is registered and used to save the modules.

$using:PackageDirectory - from another comment, this could use a module scoped variable for where these compressed/packed modules will be extracted into.

austoonz · 2025-07-18T20:09:40Z

powershell-runtime/source/modules/Private/Set-PSModulePath.ps1

        '/opt/modules', # User supplied modules as part of Lambda Layers
        [System.IO.Path]::Combine($env:LAMBDA_TASK_ROOT, 'modules') # User supplied modules as part of function package
    ) -join ':'
+    If (Test-RuntimePackedModule -Combined) {


If the paths to the extracted modules was the module scoped variable, we could just add that to the $env:ModulePath directly (perhaps if it exists, or just add it). rather than traversing the file system again.

…r `Set-PSModulePath` to use it

…eration of files/directories

First pass at implementing packed module support

bdbdf43

sean-r-williams mentioned this pull request Jun 24, 2025

Compress runtime/modules to reduce deployment package unzipped size #37

Open

sean-r-williams marked this pull request as ready for review July 18, 2025 18:35

sean-r-williams added 2 commits July 18, 2025 11:36

Merge branch 'main' of https://github.com/awslabs/aws-lambda-powershe…

642197a

…ll-runtime into sean-r-williams/37_compressed-modules

Adjust log preamble for Import-Module*

98a8765

austoonz requested changes Jul 18, 2025

View reviewed changes

sean-r-williams added 10 commits July 18, 2025 14:45

Remove alias

9963b94

Sequester output of New-Item

917c9ad

Memoize un/packed compressed module paths within module base, refacto…

db51cda

…r `Set-PSModulePath` to use it

Use script-scope unpack dirs in Import-Module*

d2fb0e9

Use module-scoped packed module paths in Import-ModuleArchive

9d6ce81

Use [directory]::CreateDirectory() instead of New-Item

dd62835

Use hashtable for defining packed module directories

d0c3297

More alias removal

a164779

Rename Test-RuntimePackedModule to Find-RuntimePackedModule

2811626

Refactor Import-Module* and Find-RuntimePackedModule to avoid re-enum…

3f80377

…eration of files/directories

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add support for packed modules to keep deployments <250mb #38

Add support for packed modules to keep deployments <250mb #38

Uh oh!

sean-r-williams commented Jun 24, 2025 •

edited

Loading

Uh oh!

sean-r-williams commented Jun 24, 2025

Uh oh!

sean-r-williams commented Jul 18, 2025 •

edited

Loading

Uh oh!

austoonz left a comment

Uh oh!

austoonz Jul 18, 2025

Uh oh!

austoonz Jul 18, 2025

Uh oh!

sean-r-williams Jul 19, 2025

Uh oh!

Uh oh!

austoonz Jul 18, 2025

Uh oh!

sean-r-williams Jul 18, 2025

Uh oh!

Uh oh!

austoonz Jul 18, 2025

Uh oh!

austoonz Jul 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Add support for packed modules to keep deployments <250mb #38

Are you sure you want to change the base?

Add support for packed modules to keep deployments <250mb #38

Uh oh!

Conversation

sean-r-williams commented Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sean-r-williams commented Jun 24, 2025

Uh oh!

sean-r-williams commented Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

austoonz left a comment

Choose a reason for hiding this comment

Uh oh!

austoonz Jul 18, 2025

Choose a reason for hiding this comment

Uh oh!

austoonz Jul 18, 2025

Choose a reason for hiding this comment

Uh oh!

sean-r-williams Jul 19, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

austoonz Jul 18, 2025

Choose a reason for hiding this comment

Uh oh!

sean-r-williams Jul 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

austoonz Jul 18, 2025

Choose a reason for hiding this comment

Uh oh!

austoonz Jul 18, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

sean-r-williams commented Jun 24, 2025 •

edited

Loading

sean-r-williams commented Jul 18, 2025 •

edited

Loading