Database Packaging

[isc-dchui]

Description

Resolves #986

Overview

• The feature adds a new lifecycle class, %IPM.Lifecycle.Database, that sits alongside the existing %IPM.Lifecycle.Module. It inherits from %IPM.Lifecycle.Base directly — not from Module — because Module has PACKAGING as a [Final] parameter and can't be subclassed to change it.
• Two new commands are exposed in %IPM.Main:
  • package-database → forces %IPM.Lifecycle.Database, routes to the Package phase
  • publish-database → same, but routes to Publish (which calls Package first, then uploads to registry)
• The -swap-db modifier is added to load/install/update to authorize the DB swap during install.

Key Details

• Package = IRIS.DAT + metadata, not source. The .tgz contains the binary IRIS.DAT, an enriched module.xml (with database and a SHA-256 ), a deps/ directory of dependency manifests, and any non-compiled resource files (wheels, FileCopy, CPF). Nothing is recompiled at install time.
• Remap-before-dismount during packaging. When building the package, the namespace is remapped to a fresh empty temp DB before the old DB is dismounted. This avoids any window where the namespace has no mounted routines DB.
• Generated resources need special handling. Classes with Generated="true" are exported from the source DB to XML before the remap, then imported into the temp DB after. Once the old DB is dismounted, compiled generated classes are inaccessible.
• Install skips Reload and Compile entirely. IsInstallContext() checks two conditions: Packaging=database AND IRIS.DAT is present in the module root. If both are true, %Reload sets SkipInvokes=1 and returns, %Compile is a no-op, and %Activate performs the swap. Pre-Activate elements are suppressed because the packaged classes don't exist until the DB is mounted.
• Swap is a rename, not a copy. PerformDatabaseSwap dismounts the existing DB, renames the current IRIS.DAT to IRIS__.DAT as a backup, renames the packaged IRIS.DAT into place, and remounts. Renames are atomic and avoid copying multi-GB files.
• Rollback is best-effort. On any failure after the swap starts, RollbackDatabaseSwap attempts dismount → delete new IRIS.DAT → rename backup back → remount. It never throws — it logs warnings and continues since it's called from a catch block.
• Update step seeding on fresh install. %Activate calls HandleAllUpdateSteps(..., seedOnly=1) on a fresh install so a later update only runs steps introduced since that version. Skipped when params("Update")=1 (an actual upgrade).
• One database package per namespace. ValidateBeforeSwap checks %IPM_Storage.ModuleItem for any other module with Packaging='database' and rejects the install if one exists.
• Python deps are on by default for package-database. Unlike package (which defaults off), package-database defaults ExportPythonDependencies=1. Both requirements.txt-based deps (resolved to wheels via ExportPythonDependencies) and explicit resources are staged into the .tgz. PythonWheel install at runtime fires during Initialize phase, which has no Database.cls override.
• Non-compiled resources are handled by resource processor hooks, not %Activate. FileCopy, WebApplication, and CPF processors run via OnBeforePhase/OnAfterPhase at the Module level before the lifecycle method is called. Overriding %Activate in Database.cls doesn't suppress them.
• publish-database forces the lifecycle class. The handler sets tCommandInfo("data","Lifecycle") = "%IPM.Lifecycle.Database", then re-routes to the existing publish command handler. Base.%Publish calls ..Package(.pParams, 0) first (which populates ..Payload), then uploads to the registry. No Database.cls override of %Publish is needed.
• dependencies.xml is actually a deps/ directory of per-dependency IRIS export XML files, each loadable with $system.OBJ.Load. RegisterDependencyMetadata loads them at install time to populate IPM storage — the code is already in the swapped-in DB, so only the IPM metadata records need registering. This is much easier to handle instead of using a fragile XSLT for a giant dependencies.xml
• ValidateIPMNotInRoutinesDB is called at two points: immediately in ShellInternal (fail-fast before any work) and again inside %Package (defense-in-depth). It checks %Library.RoutineMgr.IsMapped("%IPM.Main.CLS").
• The XSLT transform (InjectDatabasePackagingTransform XData) injects and into module.xml. The checksum placeholder REPLACECHECKSUM is substituted in ObjectScript before the stylesheet is compiled, since XSLT 1.0 has no parameterized element content.
• When installing without swap-db, only installs source-packaged modules. If none exist, but database-packaged module(s) exist, informs user. When installing with swap-db, only installs database-packaged modules.
• When publishing database-packaged modules to OCI registry, sets com.intersystems.ipm.packaging manifest annotation to "database" (for backwards compatibility, source packaged modules will not have this annotation) and use <module-version>_database__<IRIS-version> tag

Testing

• All tests are in Test.PM.Integration.DatabasePackaging. The test infrastructure uses two shared lazy-initialized namespaces to avoid ~15 namespace create/teardown cycles per run:
  • SharedNS1 (TESTDBPKGNS1): simple-db-module, module-with-invokes, module-with-dependencies
  • SharedNS2 (TESTDBPKGNS2): all-resources-module, module-with-mixed-python, module-with-requirements; also has the zot ORAS registry configured
• Per-test install namespaces are created fresh and torn down in OnAfterOneTest.

Test	What it covers
TestSimplePackageAndInstall	Package → verify IRIS.DAT + module.xml content (Packaging, checksum, SystemRequirements version) → install with load -swap-db → classes callable, module in list and history
TestPackageAndInstallWithDependencies	Package dep-module + main-with-deps; install main-with-deps → main class exists, dep appears in list
TestPackageAndInstallAllResourceTypes	Smoke test: module with class, include file, generated class, WebApp, FileCopy, PythonWheel packages and installs; include constant accessible, generated class callable, WebApp created
TestPackageAndInstallWithTestResources	-include-test-resources includes Scope=test (TestHelper) and UnitTest (Test) classes; default packaging excludes both
TestPackageWithUseCurrentDbFlag	-use-current-db completes without error and produces a package file
TestPackagingAndInstallValidation	Valid install succeeds; tampered IRIS.DAT (checksum mismatch), zero-sized IRIS.DAT, and missing deps/ directory each fail validation
TestBackupAndRollback	Backup file created in routines DB directory after swap
TestUpgradeSourceToDatabase	Source-installed module upgraded via update -path -swap-db; Packaging changes to "database"; backup confirms DB swap occurred
TestUninstallDatabasePackage	Module removed from list; backup preserved; compiled classes gone from namespace
TestMixedPackaging	Database + source package coexist in same namespace; second database package in same namespace is rejected
TestPythonDependencyPackaging	module-with-mixed-python: explicit PythonWheel (lune) + requirements.txt wheel (pycparser) both included by default, excluded with -no-export-python-deps; lune importable after install. module-with-requirements: packaging wheel included by default, excluded with flag
TestInvokeBehaviorDuringDatabaseInstall	<Invoke After="Compile"> skipped during database install; <Invoke After="Activate"> runs; global markers confirm each
TestNonCompiledResourcesAppliedDuringInstall	CPF file staged in package; FileCopy target deleted before install, confirmed recreated after install — proves resource processor hooks fire independently of %Activate override
TestUpdateStepsWithDatabasePackaging	v1 fresh install seeds Step001 (not run); v2 upgrade runs Step002 once; Step001 still not run; backup count > 1 confirms second swap
TestPublishDatabaseWithORAS	publish-database + publish to zot; source install (no -swap-db) → Packaging=module; database install (-swap-db) → Packaging=database; main-with-deps install resolves dep as source; after unpublishing source tag, install without -swap-db fails with hint to use -swap-db
TestSystemRequirementsOverwrite	package-database overwrites a stale IRIS version in <SystemRequirements> with the current version

Not Yet Handled

• Mirroring

Checklist

• This branch has the latest changes from the main branch rebased or merged.
• Changelog entry added.
• Unit (zpm test -only) and integration tests (zpm verify -only) pass.
• Style matches the style guide in the contributing guide.
• Documentation has been/will be updated
  • Source controlled docs, e.g. README.md, should be included in this PR and Wiki changes should be made after this PR is merged (add an extra issue for this if needed)
• Pull request correctly renders in the "Preview" tab.

[isc-jili]

Looks great @isc-dchui ! I left some comments!

[isc-jili]

Small nit: given that there is a nearly identical comment less than 10 lines down, suggest either removing that comment or removing this section from method description

[isc-jili]

nit: suggest renaming depDir and depsDir to be more distinctive as currently their names are a bit confusing

[isc-jili]

Question: is there any info that needs to be checked in extractOutput? I don't see it being checked below. If it doesn't need to get checked, is there any value in storing it in a variable then?

[isc-jili]

Question: if we want to restore module.xml unconditionally, would it be safer to have the following in a try/catch block in case the package-database fails?

[isc-jili]

Would it be more accurate to check for a substring of the expected error that is thrown as opposed to just the status not being OK? Since what if the status is not OK for a different reason than expected?

[isc-jili]

Question: I'm noticing for print statements some are verbose and some are always printed. Just want to double check that this is the intent.
It seems that currently this is the state of print statements:

• invoke warnings: always printed
• checksum: always printed
• package path: always printed
• many progress steps: verbose-only

[isc-jili]

Does anything in this class use %syPrompt?

[isc-jili]

Nit: ExportGeneratedResources() takes ByRef params, but I don’t think the method actually uses it?

[isc-jili]

nit: param ns seems to be unused

[isc-jili]

The new methods in this new class could all benefit from params documented in the method comments as well!