fix(standard-tests): validate CONNECTION_STATUS in test_docker_image_build_and_check

[devin-ai-integration]

Related to https://github.com/airbytehq/airbyte-internal-issues/issues/16212:

• https://github.com/airbytehq/airbyte-internal-issues/issues/16212

Summary

DockerConnectorTestSuite.test_docker_image_build_and_check in
airbyte_cdk/test/standard_tests/docker_base.py had two gaps that allowed a
connector to silently pass CI even when its check command returned
CONNECTION_STATUS: FAILED:

1. Gap 1 — no CONNECTION_STATUS validation. The test only called
   run_docker_airbyte_command(..., raise_if_errors=True), which only inspects
   TRACE ERROR messages. A connector whose check exited with code 0 and
   emitted CONNECTION_STATUS: FAILED (without a trace error) passed the test.
2. Gap 2 — status: failed scenarios were skipped. When a scenario's
   acceptance-test-config.yml entry declared status: "failed", the test
   early-returned via pytest.skip(...), so invalid_config.json-style
   scenarios were never exercised at all.

This PR closes both gaps:

• The pytest.skip for expect_exception() scenarios is removed, so
  status: failed configs now actually run the containerized check.
• run_docker_airbyte_command is only asked to raise on trace errors when the
  scenario expects success; otherwise the test inspects the result directly.
• A new _assert_check_result_matches_expected_outcome helper asserts that
  exactly one CONNECTION_STATUS message was emitted and that its status
  matches the scenario's ExpectedOutcome (SUCCEEDED / FAILED, with
  ALLOW_ANY still accepting either).

The bug was originally surfaced while working on
airbyte#76373
(source-google-ads), where a DynamicStreamCheckConfig regression returned
CONNECTION_STATUS: FAILED but CI stayed green.

Test Coverage

A new unit test file unit_tests/test/test_docker_base.py covers the new
helper with 9 parametrized cases, including the two regression cases called
out above (IDs succeed_scenario_with_failed_status_raises__gap_1 and
failed_scenario_with_failed_status_passes__gap_2). These tests construct
EntrypointOutput objects directly with synthetic CONNECTION_STATUS
messages, so they exercise the assertion logic without needing Docker.

$ poetry run pytest unit_tests/test/test_docker_base.py -v
... 9 passed in 0.39s

Declarative-First Evaluation: not applicable — this change is in the CDK's
standard-test harness, not in a declarative connector manifest.

Breaking Change Evaluation: not a breaking change for connector end users. The
change tightens an internal test in the CDK's standard-test suite, which may
cause previously-passing connectors to fail CI if their check was returning
FAILED (or emitting zero/multiple CONNECTION_STATUS messages) against a
status: "succeed" scenario — but that is the intended behavior and the
reason for the fix. No connector runtime behavior is modified, so standard
semantic versioning (PATCH/MINOR, handled by the CDK's release-drafter)
applies.

This PR is marked Related to rather than Resolves because the test-harness
fix doesn't directly close the investigative issue on its own — follow-up may
be needed to confirm no connectors regress when this tightened check lands.
Happy to flip it to Resolves if the triager prefers.

Review & Testing Checklist for Human

• Confirm that tightening test_docker_image_build_and_check to assert on
  CONNECTION_STATUS is the intended behavior, and that it's acceptable
  for any in-flight connectors whose check quietly returns FAILED for
  status: "succeed" scenarios to start failing CI.
• Confirm the Related to vs Resolves choice for the linked oncall
  issue.
• Spot-check the new helper's error messages on a real run (e.g., run the
  test against a connector locally) to make sure the assertion output is
  actionable.

Notes

• The fix is intentionally minimal and keeps all existing behavior for
  ALLOW_ANY / missing-status scenarios.
• No CDK-side metadata/version bump is committed; the repo's release-drafter
  handles versioning on merge.

Link to Devin session: https://app.devin.ai/sessions/030e6c81e83a484a8cff5faa48d1eb29

---

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1776447019-fix-docker-check-status-validation#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1776447019-fix-docker-check-status-validation

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poetry-lock - Updates poetry.lock file
• /test - Runs connector tests with the updated CDK
• /prerelease - Triggers a prerelease publish with default arguments
• /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
• /poe <command> - Runs any poe command in the CDK environment

📚 Show Repo Guidance

Helpful Resources

• CDK API Reference

📝 Edit this welcome message.

[github-actions]

PyTest Results (Fast)

4 027 tests  +9   4 016 ✅ +9   7m 9s ⏱️ -31s
    1 suites ±0      11 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit bc786ac. ± Comparison against base commit 1256a1f.

[github-actions]

PyTest Results (Full)

4 030 tests  +9   4 018 ✅ +9   11m 9s ⏱️ -24s
    1 suites ±0      12 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit bc786ac. ± Comparison against base commit 1256a1f.

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 3 additional findings.

[Anatolii Yatsuk (tolik0)]

/prerelease

Prerelease Job Info

This job triggers the publish workflow with default arguments to create a prerelease.

Prerelease job started... Check job output.

✅ Prerelease workflow triggered successfully.

View the publish workflow run: https://github.com/airbytehq/airbyte-python-cdk/actions/runs/24730905870