Add KernelCI pull-lab poller, translator, and direct KCIDB submission#4
Open
nuclearcat wants to merge 1 commit into
Open
Add KernelCI pull-lab poller, translator, and direct KCIDB submission#4nuclearcat wants to merge 1 commit into
nuclearcat wants to merge 1 commit into
Conversation
Wire pullab_cloud into the KernelCI pipeline as a PULL_LABS consumer: poll kernelci-api /events for jobs scheduled to runtime pull-labs-aws-ec2, translate the PULL_LABS job_definition into a per-run test_config, invoke the existing AWS run_pipeline(), and push per-test results directly to KCIDB via kcidb-restd-rs /submit. Pipeline callback (lava_callback.py) is deliberately bypassed; tests attach to existing build_ids resolved from the maestro node tree, so checkouts/builds are not re-emitted. New code is generic Python (stdlib urllib only, no boto3 at module top level) and ships three entry points: CLI, library, and AWS Lambda handler. Configuration is env-var-driven with config.json fallback. The KCIDB_REST=https://<token>@host/submit form is supported for zero-effort reuse of an existing kci-dev configuration; explicit KCIDB_SUBMIT_URL+KCIDB_JWT takes precedence. A new vm-tests/url-kernel-boot scaffold downloads the artifacts.kernel / artifacts.modules / optional artifacts.rootfs URLs from the job definition and emits a BenchmarkAnalyzer-compatible CSV; the deployment-specific boot step is left as a BOOT_HOOK extension point. run_pipeline() now returns its summary dict (was discarded) so the poller can map per-VM pass/fail into KCIDB rows. 76 new pytest cases cover translation edge cases, KCIDB row/revision building, status mapping, KCIDB_REST parsing, credential priority, cursor store, event filtering, and build_id resolution. All 111 tests pass (35 pre-existing + 76 new). See QUICKSTART.md for the full operator walkthrough. This is not completely verified skeleton, just to start things up, so i expect follow up PRs. Signed-off-by: Denys Fedoryshchenko <denys.f@collabora.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Wire pullab_cloud into the KernelCI pipeline as a PULL_LABS consumer: poll kernelci-api /events for jobs scheduled to runtime pull-labs-aws-ec2, translate the PULL_LABS job_definition into a per-run test_config, invoke the existing AWS run_pipeline(), and push per-test results directly to KCIDB via kcidb-restd-rs /submit.
Pipeline callback (lava_callback.py) is deliberately bypassed; tests attach to existing build_ids resolved from the maestro node tree, so checkouts/builds are not re-emitted. New code is generic Python (stdlib urllib only, no boto3 at module top level) and ships three entry points: CLI, library, and AWS Lambda handler.
Configuration is env-var-driven with config.json fallback. The KCIDB_REST=https://@host/submit form is supported for zero-effort reuse of an existing kci-dev configuration; explicit KCIDB_SUBMIT_URL+KCIDB_JWT takes precedence.
A new vm-tests/url-kernel-boot scaffold downloads the artifacts.kernel / artifacts.modules / optional artifacts.rootfs URLs from the job definition and emits a BenchmarkAnalyzer-compatible CSV; the deployment-specific boot step is left as a BOOT_HOOK extension point.
run_pipeline() now returns its summary dict (was discarded) so the poller can map per-VM pass/fail into KCIDB rows.
76 new pytest cases cover translation edge cases, KCIDB row/revision building, status mapping, KCIDB_REST parsing, credential priority, cursor store, event filtering, and build_id resolution. All 111 tests pass (35 pre-existing + 76 new).
See QUICKSTART.md for the full operator walkthrough.
This is not completely verified skeleton, just to start things up, so i expect follow up PRs.