ci: add quality gates for coverage and type checking#427
ci: add quality gates for coverage and type checking#427raballew wants to merge 4 commits intojumpstarter-dev:mainfrom
Conversation
✅ Deploy Preview for jumpstarter-docs ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
Warning Rate limit exceeded
Your organization is not enrolled in usage-based pricing. Contact your admin to enable usage-based pricing to continue reviews beyond the rate limit, or try again in 25 minutes and 21 seconds. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughAdded a CI type-check job that runs when Python paths change, added a PR-only diff-cover step to enforce 80% changed-lines coverage, and updated Python dev tooling to include diff-cover and enable branch coverage in coverage config. Changes
Sequence Diagram(s)(No sequence diagrams generated — changes are CI/workflow and tooling adjustments.) Estimated code review effort🎯 2 (Simple) | ⏱️ ~12 minutes Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
… checking Enable branch coverage measurement, add diff-cover to enforce >=80% coverage on changed lines in PRs, and add ty type checking to the lint workflow (non-blocking until existing diagnostics are resolved). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
raballew
force-pushed
the
040-quality-gates-v2
branch
from
2bd2321 to
b6c2149
Compare
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.github/workflows/python-tests.yaml:
- Around line 103-107: The current workflow step "Check coverage on changed
lines" uses the glob packages/*/coverage.xml which only matches packages that
already emit XML, so changed packages without --cov-report=xml are skipped;
modify the job to first compute the set of changed packages (git diff
--name-only origin/${{ github.base_ref }}...HEAD), then for each changed package
check for packages/<pkg>/coverage.xml and for any missing run that package's
tests to produce an XML report (e.g., run pytest or the package's test script
with --cov-report=xml) before invoking diff-cover, or alternatively fail the job
with a clear message if a changed package cannot produce a coverage XML; update
the step that runs diff-cover to pass only the discovered coverage.xml files
rather than the static glob.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 4c49de4b-965b-4e2a-a494-9ebd5413583d
📒 Files selected for processing (1)
.github/workflows/python-tests.yaml
.github/workflows/python-tests.yaml
Outdated
| - name: Check coverage on changed lines | ||
| if: github.event_name == 'pull_request' | ||
| working-directory: python | ||
| run: | | ||
| uv run diff-cover packages/*/coverage.xml --compare-branch=origin/${{ github.base_ref }} --fail-under=80 |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
BASE_REF="origin/${GITHUB_BASE_REF:-main}"
echo "Using base ref: ${BASE_REF}"
changed_packages="$(
git diff --name-only "${BASE_REF}"...HEAD \
| sed -nE 's#^python/packages/([^/]+)/.*#\1#p' \
| sort -u
)"
if [ -z "${changed_packages}" ]; then
echo "No changed python packages detected in this diff."
exit 0
fi
echo "Changed packages:"
printf ' - %s\n' ${changed_packages}
echo
echo "Checking for XML coverage config marker (--cov-report=xml) in each changed package pyproject.toml:"
while IFS= read -r pkg; do
pyproject="python/packages/${pkg}/pyproject.toml"
if [ ! -f "${pyproject}" ]; then
echo "MISS ${pkg}: pyproject.toml not found"
continue
fi
if rg -n --fixed-strings -- '--cov-report=xml' "${pyproject}" >/dev/null; then
echo "OK ${pkg}: has --cov-report=xml"
else
echo "MISS ${pkg}: missing --cov-report=xml"
fi
done <<< "${changed_packages}"Repository: jumpstarter-dev/jumpstarter
Length of output: 158
🏁 Script executed:
#!/bin/bash
# Find all packages in python/packages directory
echo "=== Python packages structure ==="
ls -la python/packages/ 2>/dev/null | head -20 || echo "Directory check failed"
echo
echo "=== All package directories ==="
find python/packages -maxdepth 1 -type d ! -name packages | sort
echo
echo "=== Checking pyproject.toml for coverage config ==="
find python/packages -maxdepth 2 -name "pyproject.toml" -type f | while read pf; do
pkg=$(echo "$pf" | cut -d/ -f3)
if grep -q "cov-report=xml" "$pf"; then
echo "OK $pkg: has cov-report=xml"
else
echo "MISS $pkg: missing cov-report=xml or coverage config"
fi
done | sortRepository: jumpstarter-dev/jumpstarter
Length of output: 7064
🏁 Script executed:
#!/bin/bash
# Check the context around lines 103-107 in the workflow file
echo "=== Workflow file context (lines 95-115) ==="
sed -n '95,115p' .github/workflows/python-tests.yaml
echo
echo "=== Full diff-cover command validation ==="
echo "Searching for all 'diff-cover' mentions in workflow:"
grep -n "diff-cover" .github/workflows/python-tests.yamlRepository: jumpstarter-dev/jumpstarter
Length of output: 1027
Guard against incomplete diff-cover coverage for changed packages.
At Line 107, packages/*/coverage.xml only includes reports that exist. Of 47 packages, only 17 have --cov-report=xml configured; the remaining 30 lack coverage XML output. If any of those 30 packages are modified, their changed lines will not be checked by the diff-cover gate.
Proposed hardening
- name: Check coverage on changed lines
if: github.event_name == 'pull_request'
working-directory: python
run: |
- uv run diff-cover packages/*/coverage.xml --compare-branch=origin/${{ github.base_ref }} --fail-under=80
+ mapfile -t changed_packages < <(
+ git diff --name-only "origin/${{ github.base_ref }}"...HEAD \
+ | sed -nE 's#^python/packages/([^/]+)/.*#\1#p' \
+ | sort -u
+ )
+
+ for pkg in "${changed_packages[@]}"; do
+ test -f "packages/${pkg}/coverage.xml" || {
+ echo "Missing coverage.xml for changed package: ${pkg}"
+ exit 1
+ }
+ done
+
+ uv run diff-cover packages/*/coverage.xml --compare-branch=origin/${{ github.base_ref }} --fail-under=80🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In @.github/workflows/python-tests.yaml around lines 103 - 107, The current
workflow step "Check coverage on changed lines" uses the glob
packages/*/coverage.xml which only matches packages that already emit XML, so
changed packages without --cov-report=xml are skipped; modify the job to first
compute the set of changed packages (git diff --name-only origin/${{
github.base_ref }}...HEAD), then for each changed package check for
packages/<pkg>/coverage.xml and for any missing run that package's tests to
produce an XML report (e.g., run pytest or the package's test script with
--cov-report=xml) before invoking diff-cover, or alternatively fail the job with
a clear message if a changed package cannot produce a coverage XML; update the
step that runs diff-cover to pass only the discovered coverage.xml files rather
than the static glob.
Packages without test coverage XML are now surfaced as GitHub Actions warnings instead of silently skipping. Also handles the case where no coverage.xml files exist at all. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Use PYTEST_ADDOPTS to ensure all packages produce coverage.xml, not just the ones that inherit the root pytest config. This closes the gap where changed lines in 30 packages were silently skipped by diff-cover. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 participant
Summary
branch = truein[tool.coverage.run]) to catch untestedif/elsepathscontinue-on-error: true) to surface type errors before merge -- can be made blocking once the 49 existing diagnostics are resolveddiff-coverto dev dependenciesRationale
As AI-generated code contributions increase, automated quality gates that prevent regression are essential for maintaining confidence when accepting changes. These three changes are the highest-impact, lowest-effort improvements:
if/elseis tested -- common in AI-generated error handling paths.Test plan
fetch-depth: 0anddiff-coverstep🤖 Generated with Claude Code