feat(testselector): pre-filter test cases by target OS before deployment#4459
Open
johnsongeorge-w wants to merge 2 commits into
Open
feat(testselector): pre-filter test cases by target OS before deployment#4459johnsongeorge-w wants to merge 2 commits into
johnsongeorge-w wants to merge 2 commits into
Conversation
johnsongeorge-w
requested review from
LiliDeng,
anirudhrb,
mcgov,
paxue,
pupacha,
rlmenge,
umfranci and
vyadavmsft
as code owners
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an opportunistic OS-based pre-filter to the test selection pipeline so that cases whose supported_os / unsupported_os metadata cannot match the target image can be dropped before environment deployment, reducing cost/time from runtime-only skips.
Changes:
- Introduces
lisa.util.os_resolverto infer a targetOperatingSystemsubclass from runbook image variables. - Extends
select_testcases()with an optionaltarget_osparameter and applies an OS compatibility pre-filter. - Wires the inferred target OS into both the runner (
lisa_runner) andlisa list --type case, and adds OS metadata to multiple suites/cases.
Reviewed changes
Copilot reviewed 27 out of 27 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| selftests/test_distro_prefilter.py | Adds unit tests for OS alias resolution, image inference, OS compatibility logic, and selector integration. |
| lisa/util/os_resolver.py | New module to resolve OS classes from aliases and infer OS from image-related variables. |
| lisa/testselector.py | Adds OS compatibility + prefilter logic and a target_os parameter to select_testcases(). |
| lisa/runners/lisa_runner.py | Infers a target OS from runbook variables and passes it into select_testcases(). |
| lisa/commands.py | Applies the same target OS inference to lisa list --type case output. |
| lisa/microsoft/testsuites/xdp/performance.py | Adds suite-level supported_os metadata and keeps runtime guard as defense-in-depth. |
| lisa/microsoft/testsuites/xdp/functional.py | Adds suite-level supported_os metadata and keeps runtime guard as defense-in-depth. |
| lisa/microsoft/testsuites/vm_extensions/waagent.py | Adds unsupported_os metadata for FreeBSD to enable pre-filtering. |
| lisa/microsoft/testsuites/rust_vmm_mshv/rust_vmm_mshv_test.py | Adds suite-level supported_os=[Linux] metadata. |
| lisa/microsoft/testsuites/power/stress.py | Adds suite-level supported_os=[Linux] metadata. |
| lisa/microsoft/testsuites/power/power.py | Adds suite-level supported_os=[Linux] metadata. |
| lisa/microsoft/testsuites/power/common.py | Adds/updates defense-in-depth comment near OS runtime skip logic. |
| lisa/microsoft/testsuites/mshv/mshv_secure_boot.py | Adds suite-level supported_os=[CBLMariner] metadata. |
| lisa/microsoft/testsuites/mshv/mshv_root_tests.py | Adds suite-level supported_os=[CBLMariner] metadata and retains runtime guard. |
| lisa/microsoft/testsuites/libvirt/libvirt_tck.py | Adds suite-level supported_os=[Ubuntu, CBLMariner] metadata and retains runtime guard. |
| lisa/microsoft/testsuites/kvm/kvm_unit_tests.py | Adds suite-level supported_os metadata and keeps runtime OS checks. |
| lisa/microsoft/testsuites/kdump/kdumpcrash.py | Adds suite-level supported_os=[Linux] metadata and keeps runtime guard. |
| lisa/microsoft/testsuites/gpu/gpusuite.py | Adds suite-level supported_os=[Linux] metadata and keeps runtime guard. |
| lisa/microsoft/testsuites/firewalld/firewalldsuite.py | Adds suite-level supported_os=[CBLMariner] metadata. |
| lisa/microsoft/testsuites/dpdk/dpdksuite.py | Adds suite-level unsupported_os=[BSD, Windows] metadata and adds case-level unsupported_os for one case. |
| lisa/microsoft/testsuites/cvm/cvm_boot.py | Adds suite-level supported_os=[CBLMariner] metadata. |
| lisa/microsoft/testsuites/cvm/cvm_azure_host.py | Adds suite-level supported_os=[CBLMariner] metadata and refines runtime skip flow. |
| lisa/microsoft/testsuites/cvm/cvm_attestation.py | Adds suite-level supported_os=[Ubuntu, CBLMariner] metadata. |
| lisa/microsoft/testsuites/core/sched_core.py | Tightens suite supported_os from Linux to CBLMariner to match runtime constraints. |
| lisa/microsoft/testsuites/core/boot.py | Adds case-level supported_os metadata for the debug-kernel boot test. |
| lisa/microsoft/testsuites/core/azure_image_standard.py | Adds OS constraints to multiple testcases to enable earlier filtering. |
| lisa/microsoft/testsuites/cloud_hypervisor/ch_tests.py | Adds suite-level supported_os=[CBLMariner, Ubuntu] metadata. |
❌ AI Test Selection — FAILED121 test case(s) selected (view run) Marketplace image: suse sles-15-sp6 gen2 latest
Test case details
|
johnsongeorge-w
force-pushed
the
johgeorg/feat_distro-prefilter_190426_083843
branch
from
09853e6 to
df6b5b6
Compare
❌ AI Test Selection — FAILED121 test case(s) selected (view run) Marketplace image: suse sles-15-sp6 gen2 latest
Test case details
|
pupacha
reviewed
May 4, 2026
johnsongeorge-w
force-pushed
the
johgeorg/feat_distro-prefilter_190426_083843
branch
from
df6b5b6 to
b039c67
Compare
❌ AI Test Selection — FAILED121 test case(s) selected (view run) Marketplace image: suse sles-15-sp6 gen2 latest
Test case details
|
Collaborator
|
johnsongeorge-w
force-pushed
the
johgeorg/feat_distro-prefilter_190426_083843
branch
from
c51785b to
3ae20d1
Compare
|
| Count | |
|---|---|
| ✅ Passed | 53 |
| ❌ Failed | 5 |
| ⏭️ Skipped | 9 |
| Total | 67 |
Test case details
| Test Case | Status | Time (s) | Message |
|---|---|---|---|
| verify_swap (lisa_0_9) | ✅ PASSED | 12.243 | |
| verify_resource_disk_io (lisa_0_10) | ✅ PASSED | 17.420 | |
| verify_resource_disk_mounted (lisa_0_8) | ✅ PASSED | 16.015 | |
| verify_scsi_disk_controller_type (lisa_0_11) | ✅ PASSED | 11.718 | |
| verify_disks_device_timeout_setting (lisa_0_7) | ✅ PASSED | 31.031 | |
| verify_os_partition_identifier (lisa_0_13) | ✅ PASSED | 4.584 | |
| verify_nvme_disk_controller_type (lisa_0_12) | ✅ PASSED | 11.223 | |
| verify_hot_add_disk_serial (lisa_0_14) | ✅ PASSED | 256.010 | |
| verify_hot_add_disk_parallel (lisa_0_17) | ✅ PASSED | 82.043 | |
| verify_hot_add_disk_serial_premium_ssd (lisa_0_16) | ✅ PASSED | 138.991 | |
| verify_hot_add_disk_serial_standard_ssd (lisa_0_15) | ✅ PASSED | 307.815 | |
| verify_hot_add_disk_parallel_standard_ssd (lisa_0_18) | ✅ PASSED | 95.118 | |
| verify_hot_add_disk_serial_random_lun_premium_ssd (lisa_0_20) | ✅ PASSED | 165.878 | |
| verify_hot_add_disk_parallel_premium_ssd (lisa_0_21) | ✅ PASSED | 123.862 | |
| verify_hot_add_disk_serial_random_lun_standard_ssd (lisa_0_19) | ✅ PASSED | 322.847 | |
| verify_nfsv4_basic (lisa_0_22) | ✅ PASSED | 205.902 | |
| verify_application_health_extension (lisa_0_77) | ✅ PASSED | 188.485 | |
| verify_azsecpack (lisa_0_76) | ✅ PASSED | 173.607 | |
| verify_azuremonitoragent_linux (lisa_0_80) | ✅ PASSED | 126.582 | |
| verify_l3_cache (lisa_0_39) | ✅ PASSED | 5.396 | |
| verify_cpu_count (lisa_0_40) | ✅ PASSED | 2.758 | |
| verify_vmbus_interrupts (lisa_0_41) | ✅ PASSED | 3.536 | |
| verify_dhcp_client_timeout (lisa_0_37) | ✅ PASSED | 5.547 | |
| verify_dns_name_resolution (lisa_0_42) | ✅ PASSED | 5.931 | |
| verify_dns_name_resolution_after_upgrade (lisa_0_43) | ✅ PASSED | 162.030 | |
| verify_floppy_module_is_blacklisted (lisa_0_68) | ✅ PASSED | 3.402 | |
| verify_initrd_modules (lisa_0_45) | ✅ PASSED | 4.115 | |
| verify_hyperv_modules (lisa_0_46) | ⏭️ SKIPPED | 7.162 | skipped: Hyper-V drivers are statically built into the kernel |
| verify_lis_modules_version (lisa_0_44) | ⏭️ SKIPPED | 0.263 | skipped: SLES not supported. This test case only supports Redhat distros. |
| verify_enable_kprobe (lisa_0_57) | ✅ PASSED | 5.172 | |
| verify_kvp (lisa_0_48) | ✅ PASSED | 10.442 | |
| verify_vm_assess_patches (lisa_0_72) | ⏭️ SKIPPED | 1.882 | skipped: Unsupported system: 'SUSE Linux Enterprise Server 15 SP6'. Linux Patch Extension doesn't support this Distro ve |
| verify_vm_install_patches (lisa_0_73) | ⏭️ SKIPPED | 0.240 | skipped: Unsupported system: 'SUSE Linux Enterprise Server 15 SP6'. Linux Patch Extension doesn't support this Distro ve |
| verify_vmbus_devices_channels (lisa_0_51) | ✅ PASSED | 11.381 | |
| verify_hyperv_platform_id (lisa_0_66) | ⏭️ SKIPPED | 25.647 | skipped: Cannot find package msr-tools or rdmsr binary |
| verify_azure_network_watcher (lisa_0_75) | ✅ PASSED | 122.438 | |
| verify_existing_script_run (lisa_0_95) | ✅ PASSED | 63.076 | |
| verify_custom_script_run (lisa_0_96) | ✅ PASSED | 63.090 | |
| verify_private_uri_script_run_failed (lisa_0_101) | ❌ FAILED | 0.539 | failed. ClientAuthenticationError: Server failed to authenticate the request. Please refer to the information in the www |
| verify_sas_uri_script_run (lisa_0_102) | ❌ FAILED | 0.608 | failed. ClientAuthenticationError: Server failed to authenticate the request. Please refer to the information in the www |
| verify_script_run_with_named_parameter (lisa_0_97) | ✅ PASSED | 62.961 | |
| verify_script_run_with_unnamed_parameter (lisa_0_98) | ✅ PASSED | 63.124 | |
| verify_script_run_with_protected_parameter (lisa_0_99) | ✅ PASSED | 63.108 | |
| verify_script_run_with_timeout_failed (lisa_0_104) | ✅ PASSED | 62.873 | |
| verify_pmu_disabled_for_arm64 (lisa_0_63) | ⏭️ SKIPPED | 3.690 | skipped: This test case does not support CpuArchitecture.X64. This validation is only for ARM64. |
| verify_timedrift_corrected (lisa_0_64) | ✅ PASSED | 51.865 | |
| verify_timesync_ptp (lisa_0_58) | ✅ PASSED | 4.479 | |
| verify_timesync_unbind_clocksource (lisa_0_59) | ✅ PASSED | 67.642 | |
| verify_timesync_unbind_clockevent (lisa_0_60) | ✅ PASSED | 2.408 | |
| verify_timesync_ntp (lisa_0_61) | ⏭️ SKIPPED | 0.292 | skipped: The distro SLES 15.6.0 doesn't support ntp, because the ntp package is no longer supported and it is implemente |
| verify_timesync_chrony (lisa_0_62) | ✅ PASSED | 8.714 | |
| verify_valid_password_run (lisa_0_119) | ✅ PASSED | 70.736 | |
| verify_vdso (lisa_0_38) | ✅ PASSED | 132.140 | |
| verify_vm_resize_increase (lisa_0_55) | ✅ PASSED | 171.527 | |
| verify_vm_resize_decrease (lisa_0_56) | ✅ PASSED | 199.094 | |
| verify_vm_hot_resize (lisa_0_53) | ✅ PASSED | 698.823 | |
| verify_vm_hot_resize_decrease (lisa_0_54) | ✅ PASSED | 697.998 | |
| verify_vmsnapshot_extension (lisa_0_69) | ✅ PASSED | 61.822 | |
| verify_exclude_disk_support_restore_point (lisa_0_70) | ✅ PASSED | 184.719 | |
| verify_gdb (lisa_0_65) | ✅ PASSED | 34.745 | |
| verify_mshv_install_succeeds (lisa_0_82) | ⏭️ SKIPPED | 0.268 | skipped: Requires a path to MSHV binaries to be passed via mshv_binpath |
| verify_base64_script_with_command_run (lisa_0_88) | ✅ PASSED | 63.430 | |
| verify_private_script_without_sas_run_failed (lisa_0_91) | ❌ FAILED | 0.566 | failed. ClientAuthenticationError: Server failed to authenticate the request. Please refer to the information in the www |
| verify_private_sas_script_run (lisa_0_93) | ❌ FAILED | 0.857 | failed. ClientAuthenticationError: Server failed to authenticate the request. Please refer to the information in the www |
| verify_vm_extension_install_uninstall (lisa_0_74) | ⏭️ SKIPPED | 0.321 | skipped: Required runbook variable(s) are missing or empty: extension_publisher='', extension_type='', extension_version |
| verify_private_sas_script_run (lisa_0_117) | ❌ FAILED | 0.557 | failed. ClientAuthenticationError: Server failed to authenticate the request. Please refer to the information in the www |
| verify_base64_script_with_command_run_failed (lisa_0_112) | ✅ PASSED | 65.085 |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 31 out of 31 changed files in this pull request and generated 5 comments.
Comments suppressed due to low confidence (1)
lisa/runners/lisa_runner.py:47
- The return type of
_resolve_target_osisOptional[type], which loses type information and makes it easy to pass a non-OperatingSystemtype intoselect_testcases(target_os=...). Please tighten this toOptional[Type[OperatingSystem]](and update the parameter type similarly) to matchinfer_target_os()andselect_testcases()signatures.
def _resolve_target_os(
variables: Any,
) -> Optional[type]:
"""Return the target OS class if distro pre-filtering is enabled.
❌ AI Test Selection — FAILED118 test case(s) selected (view run) Marketplace image: suse sles-15-sp6 gen2 latest
Test case details
|
Add a distro pre-filter to select_testcases() that drops cases whose supported_os/unsupported_os makes them inapplicable to the target distro before deployment. This avoids deploying VMs only to skip unrelated cases at runtime, saving time and cost for partner runs targeting a single distro. Target OS is inferred from image variables (marketplace_image, shared_gallery_image, vhd) using a new lisa.util.os_resolver helper. The OS alias dictionary includes both distro names and publisher names (canonical, openlogic, microsoftcblmariner, etc.) so inference works from either substring. Azure blob URL domain suffixes (.blob.core.windows.net) are stripped before matching to avoid false 'windows' hits. A runbook variable 'enable_distro_pre_filtering' (default: true) controls whether the pre-filter is active. When set to false, all test cases are kept regardless of image. Compatibility uses bidirectional issubclass so a case requiring Linux still matches target Ubuntu, and a case requiring CBLMariner is kept when the target is broader. Wires the parameter through LisaRunner._initialize and the lisa list command so previewed and executed selections agree. Includes 25+ unit tests covering alias resolution, image inference (marketplace, VHD, shared gallery), hierarchy matching, and end-to-end selection. Declares supported_os or unsupported_os metadata on 15 test suites that already perform isinstance + SkippedException checks at runtime, enabling the pre-filter to act on them. In-method guards kept as defense-in-depth for cases where OS detection misclassifies the node. Suite-level additions: - cloud_hypervisor: supported_os=[CBLMariner, Ubuntu] - cvm_attestation: supported_os=[Ubuntu, CBLMariner] - cvm_boot: supported_os=[CBLMariner] - mshv_secure_boot: supported_os=[CBLMariner] - dpdk: unsupported_os=[BSD, Windows] Per-test additions in azure_image_standard: - verify_network_manager_not_installed: supported_os=[Fedora] - verify_network_file_configuration: supported_os=[Fedora, CBLMariner] - verify_ifcfg_eth0: supported_os=[Fedora] - verify_udev_rules_moved: supported_os=[CoreOs, Fedora, CBLMariner] - verify_dhcp_file_configuration: supported_os=[Suse, CBLMariner] - verify_yum_conf: supported_os=[Fedora] - verify_hv_kvp_daemon_installed: supported_os=[Debian, CBLMariner] - verify_cloud_init_error_status: supported_os=[CBLMariner] - verify_essential_kernel_modules: supported_os=[Linux] Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Align test case/suite metadata with the runtime isinstance + SkippedException guards so the distro pre-filter can drop inapplicable cases before VM deployment.
LiliDeng
force-pushed
the
johgeorg/feat_distro-prefilter_190426_083843
branch
from
3ae20d1 to
4152329
Compare
❌ AI Test Selection — FAILED118 test case(s) selected (view run) Marketplace image: suse sles-15-sp6 gen2 latest
Test case details
|
LiliDeng
approved these changes
May 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add an opportunistic distro pre-filter to
select_testcases()that drops test cases whosesupported_os/unsupported_osmetadata makes them inapplicable to the target image before VM deployment. This avoids provisioning VMs only to hit runtimeSkippedExceptionchecks, saving time and cost on partner runs targeting a single distro.How it works
lisa.util.os_resolvermodule infers the target OS from image-related runbook variables (marketplace_image,shared_gallery_image,vhd) using an alias dictionary that maps distro names and publisher names to LISA OS classes.select_testcases()accepts an optionaltarget_osparameter. When set, it uses bidirectionalissubclassto check each case'ssupported_os/unsupported_osagainst the target and drops incompatible cases.enable_distro_pre_filtering(default:true) controls whether the pre-filter is active. Set tofalseto disable.isinstance+SkippedExceptionguards at runtime now also declaresupported_os/unsupported_osin their metadata so the pre-filter can act on them.Design constraints and known limitations
supported_osmetadata +isinstanceruntime guard@requires_distrodecorator driving bothKey architectural acknowledgment: the runtime
isinstanceguards remain the authoritative enforcement. The pre-filter is a best-effort optimization that reduces wasted deployments but does not replace the runtime check. The two can drift if an author updates one but not the other — this is a known DRY trade-off accepted for v1.Validation
Tested with
lisa list --type case(no deployment) across 5 marketplace images:almalinux almalinux-arm 9-arm-gen2Canonical 0001-com-ubuntu-minimal-focal-dailyDebian debian-11 11MicrosoftCBLMariner cbl-mariner cbl-mariner-2RedHat RHEL 7_9All lint checks pass (black, flake8, mypy). 27 unit tests covering alias resolution, image inference (marketplace, VHD, shared gallery), hierarchy matching, and end-to-end selection.
Future improvements (out of scope for this PR)
isinstanceguardCo-authored-by: Copilot 175728472+Copilot@users.noreply.github.com