ci: implement nightly container-based test runs and fast local tox-based testing

[markuslf]

Goal

Split the test strategy into tiers to speed up the development workflow while maintaining real cross-platform coverage.

Tier 1: Fast local / CI tests (tox + multiple Python versions)

• Run unit tests with test data (no containers, no external services) via tox
• Cover Python 3.9 (RHEL 8) through 3.14 (current)
• Runs in seconds, suitable for every commit and local development
• Should work for all plugins that have a unit-test/run with --test data

Tier 2: Nightly container-based tests (podman, GitHub Actions)

Scheduled GitHub Actions workflow (cron, e.g. daily at 02:00 UTC). Failures are reported as notifications the next morning, not blocking PRs.

Approach:

1. Build cacheable base images per distro with Python and all dependencies (psutil, pymysql, etc.). These are built once and cached, not rebuilt every night.

2. Run ALL --test-based unit tests inside every distro container. Copy the plugin + lib into the container, run python3 plugin --test=.... This is fast (milliseconds per check, no external services needed) and tests real platform behavior - different Python versions, different psutil versions, different system libraries. This is NOT the same as just testing the lib.

3. Run plugin-specific container tests (with services) where they exist. Plugins with unit-test/containerfiles/ that start MySQL, Redis etc. run their full integration tests.

Run containers in parallel (GitHub Actions matrix) to keep total wall-clock time manageable.

Target Linux platforms (via podman, official container images)

• Arch Linux
• Debian 11, 12, 13
• Fedora (current)
• openSUSE Leap / Tumbleweed
• Photon OS (VMware, official Docker image)
• RHEL 8, 9, 10 (via UBI)
• Ubuntu 20.04, 22.04, 24.04, 26.04 (when released)

Windows

Windows is not part of the nightly tests. Plugins with a .windows file must be compiled (PyInstaller/Nuitka) before testing, and the build requires manual confirmation due to code signing. Windows testing belongs in the release/build process, not in a nightly cron job.

Not planned

• Distributions older than RHEL 8 (Python < 3.9, all EOL)
• Slackware (no official container images)
• SLES (requires subscription for images)
• ARM/Raspberry Pi (lower priority, can be added later via QEMU emulation or dedicated ARM runners)

[markuslf]

The example plugin now demonstrates the new test pattern. Key changes:

• lib/lftest.py has a new run() function for declarative, data-driven unit tests using unittest.subTest().
• Test definitions are a simple list of dicts (TESTS) instead of one method per testcase. See check-plugins/example/unit-test/run for the reference implementation.
• Test data files use descriptive names (ok-basic, warn-threshold, crit-service-down) instead of EXAMPLE01.
• Empty retc/ and stderr/ directories are no longer created.
• tools/run-unit-tests discovers and runs all plugin tests from the repo root, with optional plugin filter (python tools/run-unit-tests disk-smart).
• tox.ini is in place for multi-version testing (Python 3.9 through 3.14).
• The CONTRIBUTING has been updated with the new test structure, naming conventions, and usage examples.

Existing tests continue to work as-is. New tests should follow the declarative pattern.

[markuslf]

Two follow-ups based on the current build pipeline (.github/workflows/lf-build-linux-x86_64.yml), which builds packages for sle15 and sle16:

1. SLE 16: not mentioned in the target list. Should be added explicitly, either as a test target or as an explicit exclusion with reasoning.
2. SLES vs. openSUSE Leap: Leap is currently listed as a target. The assumption is that Leap serves as a proxy for SLE. If that assumption holds, document it. If not, SUSE BCI images (registry.suse.com/bci/bci-base) are publicly pullable without a subscription and could cover SLE 15 SP6 and SLE 16 directly, which would remove the "requires subscription" blocker for SLES.