feat: system test framework proposal#111
Open
SamBarker wants to merge 7 commits into
Open
Conversation
Introduces layered abstractions (ProxyScenario, FilterSpec, ProxyFixture, ProxyHandle) that separate test intent from deployment mechanism, enabling deployment-agnostic feature tests and test-first development. Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
OperatorCapability now models the operator's externally observable state via generation-based reconciliation observation rather than the checksum annotation. Tests that assert on specific operator mechanisms (e.g. checksum change detection) observe resource state directly. Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
OperatorCapability handles convergence and deployment agnosticism. Tests that assert on specific resource state (e.g. checksum annotations) use an injected KubernetesClient to observe resources directly, keeping the two concerns separate. Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
Three distinct test categories become separate modules with different compile-time dependencies: systemtest-feature (no K8s dependency), systemtest-operator (K8s client and CRD types), and systemtest-installer (one test per install method). All three are TCK-consumable. Installer is a public interface — the primary downstream extension point. CrdProxyFixture replaces OperatorProxyFixture to reflect that the fixture uses the CRD API, not the operator's internals. Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
Fixture/installer composition is extension-internal, driven by system properties. Installer tests use a single smoke test — the test does not vary between installers, CI provides the matrix. Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
- Summary now mentions all four test categories and the Installer - "Three categories/modules" → "Four" throughout - Add systemtest-webhook to TCK module list with description - Feature test example: remove namespace parameter, use kafkaClient - Tags section: clarify module boundaries as primary separation - Rejected alternatives: @operator tag "ensures present", not "selects fixture" - Webhook section: rewrite as two-module story (installer + behaviour) - Affected projects: add systemtest-webhook module - Fix grammar: "An CrdProxyFixture" → "A CrdProxyFixture" Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ProxyScenario) from deployment mechanism (ProxyFixture) with convergence gating (ProxyHandle)systemtest-feature,systemtest-operator,systemtest-webhook,systemtest-installerInstalleras the primary downstream extension point — downstream varies by installation method, not proxy deploymentTest plan
🤖 Generated with Claude Code