Fix test infrastructure for Python-version-excluded providers#63793
Merged
potiuk merged 3 commits intoapache:mainfrom Mar 17, 2026
Merged
Fix test infrastructure for Python-version-excluded providers#63793potiuk merged 3 commits intoapache:mainfrom
potiuk merged 3 commits intoapache:mainfrom
Conversation
Dev-iL
requested review from
amoghrajesh,
ashb,
bugraoz93,
choo121600,
ephraimbuddy,
gopidesupavan,
jason810496,
jedcunningham,
jscheffl and
potiuk
as code owners
boring-cyborg
bot
added
area:dev-tools
area:docker-tests
area:production-image
Dev-iL
force-pushed
the
2603/fix-excluded-provider-tests
branch
3 times, most recently
from
48f3621 to
c67db2a
Compare
When running Providers[google] or Providers[amazon] on Python 3.14, generate_args_for_pytest removes the test folders for excluded providers, but the skip check in _run_test only triggered when the --ignore filter itself removed something. Since the folders were already removed upstream, the guard condition was never met, leaving pytest with only flags and no test directories — causing it to crash on unrecognized custom arguments. Remove the overly strict guard so the skip fires whenever no test directories remain in the args.
The docker tests expected all providers from prod_image_installed_providers.txt to be present, but providers like google and amazon declare excluded-python-versions in their provider.yaml. On Python 3.14, these providers are correctly excluded from the PROD image at build time, but the tests didn't account for this. Read provider.yaml exclusions and filter expected providers/imports based on the Docker image's Python version.
get_distribution_specs.py now reads Requires-Python metadata from each wheel and skips wheels that are incompatible with the running interpreter. This prevents excluded providers (e.g. amazon on 3.14) from being passed to pip/uv and installed despite their exclusion. Also fix the requires-python specifier generation in packages.py: !=3.14 per PEP 440 only excludes 3.14.0, not 3.14.2. Use !=3.14.* wildcard to exclude the entire minor version.
Dev-iL
force-pushed
the
2603/fix-excluded-provider-tests
branch
from
c67db2a to
51c747a
Compare
potiuk
approved these changes
Mar 17, 2026
github-actions bot
pushed a commit
that referenced
this pull request
Mar 17, 2026
…ders (#63793) * Skip provider tests when all test directories are excluded When running Providers[google] or Providers[amazon] on Python 3.14, generate_args_for_pytest removes the test folders for excluded providers, but the skip check in _run_test only triggered when the --ignore filter itself removed something. Since the folders were already removed upstream, the guard condition was never met, leaving pytest with only flags and no test directories — causing it to crash on unrecognized custom arguments. Remove the overly strict guard so the skip fires whenever no test directories remain in the args. * Fix PROD image docker tests for Python-version-excluded providers The docker tests expected all providers from prod_image_installed_providers.txt to be present, but providers like google and amazon declare excluded-python-versions in their provider.yaml. On Python 3.14, these providers are correctly excluded from the PROD image at build time, but the tests didn't account for this. Read provider.yaml exclusions and filter expected providers/imports based on the Docker image's Python version. * Skip Python-incompatible provider wheels during PROD image build get_distribution_specs.py now reads Requires-Python metadata from each wheel and skips wheels that are incompatible with the running interpreter. This prevents excluded providers (e.g. amazon on 3.14) from being passed to pip/uv and installed despite their exclusion. Also fix the requires-python specifier generation in packages.py: !=3.14 per PEP 440 only excludes 3.14.0, not 3.14.2. Use !=3.14.* wildcard to exclude the entire minor version. (cherry picked from commit 48ec002) Co-authored-by: Dev-iL <6509619+Dev-iL@users.noreply.github.com>
Backport successfully created: v3-1-testNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
provider.yamlexclusions and filter expected providers/imports based on the image's Python version.Details
Skip provider tests when all test directories are excluded
When running provider test suites (e.g.,
Providers[google]) on a Python version where that provider is excluded,generate_args_for_pytestremoves the test folders upstream. However, the skip check in_run_testonly triggered when the--ignorefilter itself removed something. Since folders were already gone, the guard condition was never met, leaving pytest with only flags and no test directories — causing it to crash on unrecognized custom arguments.The fix removes the overly strict guard so the skip fires whenever no test directories remain in the args.
Fix PROD image docker tests for excluded providers
The docker tests expected all providers from
prod_image_installed_providers.txtto be present, but providers like google and amazon may declareexcluded-python-versionsin theirprovider.yaml. The tests now read these exclusions and filter expected providers and imports based on the Docker image's Python version.related: #63520
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Opus 4.6 following the guidelines
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.