[SPARK-56721][INFRA] Add master server vs 4.0 client CI for all PRs

[gaogaotiantian]

What changes were proposed in this pull request?

Add a task to test master server vs 4.0 client for backward compatibility.

Why are the changes needed?

We've broken old client for many times. It's a scheduled task and the problem is often found a few days later. We should catch the issue asap.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

CI should give result about whether it worked.

Was this patch authored or co-authored using generative AI tooling?

No.

[Yicong-Huang]

I like this idea! but in the rare cases where the change has to be back ported to different branches (like #55670), the CI will always fail. What's our protocol to handle it? do we just merge it with failed CI and then backport separately?

[gaogaotiantian]

The CI is intentionally enabled only on PRs (not main commits). This is supposed to be a warning like - you need to fix the client ASAP. I think the proper protocol is to have both PRs ready, and merge them sequentially in a short window.

[Yicong-Huang]

The CI is intentionally enabled only on PRs (not main commits). This is supposed to be a warning like - you need to fix the client ASAP. I think the proper protocol is to have both PRs ready, and merge them sequentially in a short window.

I think we should note this as a warning somehow. for example post a comment. because some reviewers/committers expect a green CI before review: If a PR's CI is failing, they may not spend time to review it.