Modernize GitHub Actions test runner

[Repiteo]

Cleans up and consolidates the workflows for running tests. Now everything is handled via matrix in one file. Tests across 5 GitHub-hosted runners (macos-15, ubuntu-22.04, ubuntu-24.04, windows-2022, windows-2025), each using the min/max supported Python versions (3.9-3.14). As versions below 3.9 aren't included, this could wait until #4827 is merged

Contributor Checklist:

• I have created a new test or updated the unit tests to cover the new/changed functionality.
• I have updated CHANGES.txt and RELEASE.txt (and read the README.rst).
• I have updated the appropriate documentation

[bdbaddog]

Why did you remove appveyor? It's still being used.

Looks like you took info from a bunch of files, deteled them and stuck all that logic in one big file.
I'm not sure that's an improvement...

Perhaps a bit more explanation than what's in the PR above would be helpful as to the why you made these changes?

[mwichmann]

As usual when tweaking the CI, which happens one or more times a year, it probably makes sense to think about what we need to accomplish: operating systems, python versions, architectures, build frequency. The build matrix could be massive but that's obviously impractical: Linux/Windows/Mac; x86/arm64; 3.7 (or 3.9) - 3.14 and devel; trigger all on every commit to a PR and to main or be more selective with some jobs, etc. Oldest-and-newest Python doesn't seem a terrible combo to me.

[Repiteo]

The short version is that all the tests were consolidated such that all of our bases and use-cases were accounted for without redundancies. So workflows that were not rendered redundant in testing, such as the package builder, were left alone. The consolidation invokes the following:

• Matrix setup to ensure each currently-supported GitHub runner is tested against the currently-supported max/min Python versions
• Run all detectable tests (experimental + frameworks included)
• Setup dependencies depending on the platform (linux-specific and windows-specific jobs)
• Setup file for ignoring selection of problematic tests, based on os and python version
• Running the test itself with the max number of jobs available

This includes some QOL to the job runner itself, such as:

• A concurrency group to premptively cancel the action if a more recent change was detected
• Run action on any branch, not just the default (allows forks to test changes in their own repo without needing to dispatch a workflow each time)
• Impose a timeout of 30 minutes, so any workflow caught in an infinite loop won't lockout a worker for the 6 hour default
• Utilize the most up-to-date actions available, allowing for cleaner syntax (ie: builtin pip dependency resolution)

Any tests that had their use-cases covered, or were otherwise redundant in their testing, were removed outright. This included AppVeyor, whose only real niche would be running older Visual Studio versions. Combined with the verification in the Discord that GHA has no runtime limits for public repositories, I didn't see much of a need to keep it around.

The only other change this PR were to stack-size.py to make the check function on certain Python 3.14 versions. There's still other flaky tests, but this was the one instance of something failing outright that could be easily addressed

[bdbaddog]

Does github actions give us the range of MSVC that appveyor does?

[Repiteo]

Nothing older than 2022. I don't think that's worth AppVeyor sticking around, though

[bdbaddog]

Nothing older than 2022. I don't think that's worth AppVeyor sticking around, though

It is. Ideally we'll keep testing whatever old versions appveyor has as long as they have them.
There are lots of projects out in the wild not using latest msvc's...

Ideally we'll try to at least be aware if any changes break our support of them.

[mwichmann]

Nothing older than 2022. I don't think that's worth AppVeyor sticking around, though

There's just no good answer to that particular problem. On the one hand, only 2022 and 2026 are in regular support (extended support runs out for 2017 in April '27 and 2019 in April '29). But we also know that maybe more than other compilation toolchains, people continue to use truly ancient versions.

The support picture has changed again, btw - releases starting with 2026 will be annual, only have full support until the next year (aka the next drop), and only one more year of extended support. But that's a concern for the future.

[Repiteo]

Reverted the removal of AppVeyor. Instead, the scope of it was changed to mirror GHA: only test the oldest and newest Python versions. It's now only targeting VS2017 and VS2019, the runners unsupported by GHA.

This was the only change made, so I opted for a force-push to preserve AppVeyor's commit history

[bdbaddog]

There's no python > 3.8 for MSVC 2017 image on appveyor. So you'll have to change that to Visual Studio 2019 image instead.

[bdbaddog]

On rerun most failures were fixed. Committing.