Rework startup/upgrade/bootstrap process to prune modules before core upgrade

[labkey-adam]

Rationale

Adding SNPRC modules to the TeamCity PostgreSQL distributions resulted in memory leaks since their SND dependency is not yet present.

Currently, modules are removed due to missing dependencies after the core module is upgraded. The removed modules show up as leaked, since several code paths end up holding onto them. For example, core bootstrap() creates key containers (/, /home, /shared), which causes us to enumerate the active modules, which loads file-based webparts, which creates file watcher listeners that hold onto their owning modules. While webparts are the culprit here, it's likely that other resources could cause similar leaks.

My solution is to rework the startup/upgrade/bootstrap process to prune modules with missing dependencies before core upgrade occurs. So now we prune for database compatibility and then immediately prune for module dependencies. As a result, core upgrade and init() implementations that enumerate active modules work on the pruned list. A side benefit is that we log messages about all pruned modules (regardless of reason) together. Module initialization still happens after core upgrade, and any module that fails initialization still gets pruned along with its dependents. A module leak will likely happen in this case, but I think it's rare for init() to throw, plus it's always treated as a module startup error.

I considered initializing all modules before core upgrade, which is appealing, in theory. However, that would have meant any operation that touches the database (role registration, property retrieval & update, etc.) would need to move from init() into startup() since core schemas could be empty or in a bad state when init() is invoked. This would have changed our rules and forced updates to 25+ modules that we manage (plus potentially others that we don't).

Related Pull Requests

• Allow registering multiple InvoicedItemsProcessingService impls wnprc-modules#917