Skip to content

Commit 524884f

Browse files
AkashKumar7902AkashKumar7902
committed
ci: doccano-django sample coverage gate (build vs release)
Adds .github/workflows/doccano-django.yml — runs ONLY on changes under doccano-django/ (or this workflow file) so unrelated samples in this repo don't pay the doccano runtime cost. Three jobs: * `build-coverage` — checks out the PR's HEAD ref, brings up the sample's compose, drives flow.sh bootstrap + record-traffic with a per-call audit log enabled, runs flow.sh coverage. Captures the percentage as a job output. * `release-coverage` — same end-to-end against github.event.pull_request.base.ref (typically main) so we have a baseline to compare against. Skipped on direct push events to main (no baseline to diff against — main IS the baseline). * `coverage-gate` — fails the PR if build's coverage drops more than COVERAGE_THRESHOLD pp below release. COVERAGE_THRESHOLD defaults to 1.0pp; override with the `DOCCANO_COVERAGE_THRESHOLD` actions variable per-repo. Sticky-comments the PR with the diff via marocchino/sticky-pull-request-comment so reviewers see the delta inline. The two measurement jobs share their body via .github/workflows/scripts/run-and-measure.sh — same script, different ref. Lifting it out of the YAML keeps the YAML focused on orchestration (matrix / outputs / artifacts) and the bash on the actual workflow logic. Coverage source uses flow.sh's per-call audit log (DOCCANO_FIRED_ROUTES_FILE). That makes the measurement genuinely keploy-independent: the workflow doesn't run keploy at all, doesn't compare against recorded test sets, just measures what the sample's flow.sh ACTUALLY exercises against doccano's URL resolver. Lane scripts in keploy/integrations and keploy/enterprise consume the same flow.sh but use the keploy/test-set-*/tests/*.yaml tree as their numerator (authoritative — only calls keploy actually captured count). Both modes are wired into flow.sh::doccano_list_recorded_routes via the DOCCANO_FIRED_ROUTES_FILE fallback. Sample-side changes: * flow.sh::doccano_wait_for_fixed_token extracted as its own function (was inlined into doccano_bootstrap_token, broke doccano_record_traffic's forward reference and silently fail-fasted under set -e). * flow.sh::doccano_record_traffic gates on doccano_wait_for_fixed_token before any curl fires — port-open isn't a sufficient readiness signal under SIGINT-driven shutdown, the very first curl -fsS POST would 5xx on a still-booting gunicorn and silently kill the script. * flow.sh::log_fired writes (METHOD, URL) to DOCCANO_FIRED_ROUTES_FILE before each curl in doccano_record_traffic. Cheap, optional (no-op when env var unset), and keeps the audit log adjacent to the curl that produces it so future contributors can't add a curl without also adding the log entry. * flow.sh::doccano_list_recorded_routes falls back to the audit log when no keploy/test-set-*/tests/*.yaml exists — the standalone-mode numerator the workflow needs. Verified locally: workflow body (`run-and-measure.sh`) runs end-to-end against bare doccano in ~3 minutes, captures 16 unique (method, path) pairs, emits coverage=11.1% to GITHUB_OUTPUT. The gate logic itself is plain bash + python3 arithmetic; no codecov/coveralls dependency, no hosted service needed. Signed-off-by: Akash Kumar <meakash7902@gmail.com>
1 parent d13903d commit 524884f

3 files changed

Lines changed: 361 additions & 18 deletions

File tree

‎.github/workflows/doccano-django.yml‎

Lines changed: 172 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,172 @@
1+
# doccano-django sample CI — keploy-independent end-to-end smoke +
2+
# coverage gate.
3+
#
4+
# Triggers ONLY on changes under doccano-django/ (or this workflow
5+
# file). Other samples in this repo have their own orthogonal CI;
6+
# gating the whole repo on every doccano change would slow them
7+
# all down for no benefit.
8+
#
9+
# What it gates:
10+
# * `release-coverage` — checks out the PR's base branch (main)
11+
# and runs the sample end-to-end: docker compose up, bootstrap
12+
# admin token, drive flow.sh record-traffic with the per-call
13+
# audit log enabled, capture the route-coverage percentage from
14+
# `flow.sh coverage`. This is the baseline.
15+
# * `build-coverage` — same end-to-end against the PR's HEAD ref.
16+
# * `coverage-gate` — fails the PR if `build`'s coverage drops
17+
# more than COVERAGE_THRESHOLD percentage points below
18+
# `release`. Default threshold is 1.0pp; override via repo
19+
# variable `DOCCANO_COVERAGE_THRESHOLD` for a tighter or
20+
# looser bar.
21+
#
22+
# On push to main, only `build-coverage` runs (no baseline to
23+
# compare against — main IS the baseline).
24+
#
25+
# Standards-aligned choices:
26+
# * `paths:` filter on both push and pull_request triggers — the
27+
# canonical GH Actions way to scope a workflow to one
28+
# subdirectory.
29+
# * Job outputs (steps.<id>.outputs.coverage → needs.<job>.outputs)
30+
# to thread the captured percentage between jobs.
31+
# * `concurrency:` cancel-in-progress on the same ref so a stale
32+
# run doesn't waste runner minutes.
33+
# * actions/upload-artifact for the human-readable
34+
# coverage_report.txt — reviewers can inspect missing routes
35+
# directly from the PR's "checks" tab.
36+
# * marocchino/sticky-pull-request-comment for the PR-side diff
37+
# comment. Pinned-by-header so successive runs update the same
38+
# comment instead of fanning out.
39+
# * The compare step is plain bash + python3 (no external
40+
# coverage service). For full Python coverage.py XMLs you'd
41+
# want diff-cover or codecov, but the sample's coverage is
42+
# API-route-based (single percentage), so the gate is a 3-line
43+
# subtraction.
44+
#
45+
# Sample is genuinely keploy-independent here: the workflow uses
46+
# flow.sh's $DOCCANO_FIRED_ROUTES_FILE per-call audit log as its
47+
# numerator source, not a keploy recording. The lane scripts in
48+
# keploy/integrations and keploy/enterprise consume the same
49+
# flow.sh, but use the keploy/test-set-*/tests/*.yaml tree as
50+
# their numerator (authoritative — only calls keploy actually
51+
# CAPTURED count). Both modes are wired into
52+
# `flow.sh::doccano_list_recorded_routes`.
53+
name: doccano-django sample
54+
55+
on:
56+
pull_request:
57+
paths:
58+
- 'doccano-django/**'
59+
- '.github/workflows/doccano-django.yml'
60+
push:
61+
branches: [main]
62+
paths:
63+
- 'doccano-django/**'
64+
- '.github/workflows/doccano-django.yml'
65+
workflow_dispatch: {}
66+
67+
concurrency:
68+
group: doccano-django-${{ github.ref }}
69+
cancel-in-progress: true
70+
71+
env:
72+
COVERAGE_THRESHOLD: ${{ vars.DOCCANO_COVERAGE_THRESHOLD || '1.0' }}
73+
74+
jobs:
75+
build-coverage:
76+
name: build (current ref) coverage
77+
runs-on: ubuntu-latest
78+
timeout-minutes: 20
79+
outputs:
80+
coverage: ${{ steps.measure.outputs.coverage }}
81+
steps:
82+
- uses: actions/checkout@v4
83+
- id: measure
84+
name: Run sample end-to-end + measure coverage
85+
working-directory: doccano-django
86+
env:
87+
DOCCANO_FIRED_ROUTES_FILE: ${{ runner.temp }}/fired-routes-build.log
88+
DOCCANO_PHASE: ci-build
89+
run: ../.github/workflows/scripts/run-and-measure.sh
90+
91+
- name: Upload coverage report
92+
if: always()
93+
uses: actions/upload-artifact@v4
94+
with:
95+
name: coverage-build
96+
path: doccano-django/coverage_report.txt
97+
if-no-files-found: warn
98+
99+
release-coverage:
100+
if: github.event_name == 'pull_request'
101+
name: release (base ref) coverage
102+
runs-on: ubuntu-latest
103+
timeout-minutes: 20
104+
outputs:
105+
coverage: ${{ steps.measure.outputs.coverage }}
106+
steps:
107+
- uses: actions/checkout@v4
108+
with:
109+
ref: ${{ github.event.pull_request.base.ref }}
110+
- id: measure
111+
name: Run sample end-to-end + measure coverage
112+
working-directory: doccano-django
113+
env:
114+
DOCCANO_FIRED_ROUTES_FILE: ${{ runner.temp }}/fired-routes-release.log
115+
DOCCANO_PHASE: ci-release
116+
run: ../.github/workflows/scripts/run-and-measure.sh
117+
118+
- name: Upload coverage report
119+
if: always()
120+
uses: actions/upload-artifact@v4
121+
with:
122+
name: coverage-release
123+
path: doccano-django/coverage_report.txt
124+
if-no-files-found: warn
125+
126+
coverage-gate:
127+
if: github.event_name == 'pull_request'
128+
name: coverage gate
129+
needs: [build-coverage, release-coverage]
130+
runs-on: ubuntu-latest
131+
steps:
132+
- name: Compare build vs release
133+
env:
134+
BUILD: ${{ needs.build-coverage.outputs.coverage }}
135+
RELEASE: ${{ needs.release-coverage.outputs.coverage }}
136+
THRESHOLD: ${{ env.COVERAGE_THRESHOLD }}
137+
BASE_REF: ${{ github.event.pull_request.base.ref }}
138+
run: |
139+
set -Eeuo pipefail
140+
if [ -z "${BUILD:-}" ] || [ -z "${RELEASE:-}" ]; then
141+
echo "::error::missing coverage outputs — build='${BUILD:-}' release='${RELEASE:-}'"
142+
exit 1
143+
fi
144+
drop=$(python3 -c "print(round(${RELEASE} - ${BUILD}, 2))")
145+
echo "Release (${BASE_REF}): ${RELEASE}%"
146+
echo "Build (this PR): ${BUILD}%"
147+
echo "Drop: ${drop}pp (threshold ${THRESHOLD}pp)"
148+
if python3 -c "import sys; sys.exit(0 if (${RELEASE} - ${BUILD}) > ${THRESHOLD} else 1)"; then
149+
echo "::error::doccano-django coverage dropped from ${RELEASE}% → ${BUILD}% (-${drop}pp), exceeding the ${THRESHOLD}pp threshold."
150+
echo "Suggested actions:"
151+
echo " * Add curl(s) to flow.sh::doccano_record_traffic that exercise the routes you changed/touched."
152+
echo " * If the route(s) was intentionally retired, drop it from doccano-django/flow.sh::doccano_list_routes' SCOPE_PREFIXES too so it's removed from the denominator."
153+
exit 1
154+
fi
155+
echo "OK — coverage delta within ${THRESHOLD}pp threshold."
156+
157+
- name: Sticky PR comment
158+
if: ${{ !cancelled() }}
159+
uses: marocchino/sticky-pull-request-comment@v2
160+
with:
161+
header: doccano-django-coverage
162+
message: |
163+
### doccano-django sample coverage
164+
165+
| ref | coverage |
166+
|---|---|
167+
| base (`${{ github.event.pull_request.base.ref }}`) | **${{ needs.release-coverage.outputs.coverage }}%** |
168+
| this PR | **${{ needs.build-coverage.outputs.coverage }}%** |
169+
170+
Threshold: PR may not drop coverage by more than **${{ env.COVERAGE_THRESHOLD }}pp**. Override per-repo via the `DOCCANO_COVERAGE_THRESHOLD` actions variable.
171+
172+
Coverage measures the API surface (`/v1/projects/*` + `/v1/me` + `/v1/users` + `/v1/health` + `/v1/auth`) that `flow.sh::doccano_record_traffic` actually exercises against the running backend's URL resolver. Reports are attached as artifacts on each job ("coverage-build" / "coverage-release").

‎.github/workflows/scripts/run-and-measure.sh‎

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
#!/usr/bin/env bash
2+
#
3+
# run-and-measure.sh — bring doccano up via the sample's compose,
4+
# run flow.sh bootstrap + record-traffic with the per-call audit
5+
# log enabled, run flow.sh coverage, and emit `coverage=PCT`
6+
# onto $GITHUB_OUTPUT for the downstream coverage-gate job.
7+
#
8+
# Called from .github/workflows/doccano-django.yml's
9+
# build-coverage and release-coverage jobs (one per ref under
10+
# comparison). Both jobs source the same script so the
11+
# measurement is identical across refs — any drift in the
12+
# numerator definition would otherwise produce a misleading
13+
# delta.
14+
#
15+
# Inputs (all from the workflow env):
16+
# DOCCANO_FIRED_ROUTES_FILE — per-call audit log path; passed
17+
# through to flow.sh so its
18+
# record-traffic loop logs each
19+
# (METHOD, URL) pair, and so its
20+
# coverage subcommand uses that
21+
# file as the standalone
22+
# numerator.
23+
# DOCCANO_PHASE — label spliced into the project
24+
# name so build vs. release runs
25+
# don't collide on volume names
26+
# (compose project naming inside
27+
# the GH runner is per-job
28+
# anyway, but DOCCANO_PHASE shows
29+
# up in the test fixtures and
30+
# is useful for diffing logs).
31+
# GITHUB_OUTPUT — standard GH Actions sink for
32+
# step outputs.
33+
set -Eeuo pipefail
34+
35+
export DOCCANO_BACKEND_CONTAINER="${DOCCANO_BACKEND_CONTAINER:-doccano_backend}"
36+
export DOCCANO_DB_CONTAINER="${DOCCANO_DB_CONTAINER:-doccano_db}"
37+
export DOCCANO_APP_PORT="${DOCCANO_APP_PORT:-18080}"
38+
export DOCCANO_FIXED_TOKEN="${DOCCANO_FIXED_TOKEN:-ac38262065f0ae1476b6a707d9d697a101764a6b}"
39+
: "${DOCCANO_FIRED_ROUTES_FILE:?DOCCANO_FIRED_ROUTES_FILE must be set by the workflow}"
40+
41+
# Reset audit log for this run; otherwise a prior run's entries
42+
# would inflate the numerator on a re-trigger.
43+
: >"$DOCCANO_FIRED_ROUTES_FILE"
44+
45+
# Stage 1: bring up doccano with bootstrap so the admin user +
46+
# fixed token persist into the named volume.
47+
DOCCANO_SKIP_BOOTSTRAP=0 docker compose up -d
48+
49+
# Wait for the backend to start serving (not just port-open).
50+
# Cold doccano boot runs Django migrations + admin user create,
51+
# which on a GH runner can hit 90-120s.
52+
for i in $(seq 1 120); do
53+
code=$(curl -sS -o /dev/null -w '%{http_code}' \
54+
"http://127.0.0.1:${DOCCANO_APP_PORT}/v1/health/" 2>/dev/null || echo "")
55+
if [ -n "$code" ] && [ "$code" != "000" ]; then break; fi
56+
sleep 2
57+
done
58+
59+
bash flow.sh bootstrap 240
60+
docker compose down --remove-orphans
61+
62+
# Stage 2: re-launch in skip-bootstrap mode against the populated
63+
# volume — same shape the keploy lanes use.
64+
DOCCANO_SKIP_BOOTSTRAP=1 docker compose up -d
65+
66+
# Drive traffic. flow.sh::doccano_record_traffic gates on
67+
# doccano_wait_for_fixed_token internally, so this won't fire
68+
# curls at a half-booted backend.
69+
bash flow.sh record-traffic
70+
71+
# Coverage report — uses DOCCANO_FIRED_ROUTES_FILE as numerator
72+
# since no keploy/test-set-* tree exists in the standalone case.
73+
COVERAGE_REPORT_FILE="$PWD/coverage_report.txt" bash flow.sh coverage
74+
75+
# Pull the percentage out of the report's `Covered N/M (XX.X%)`
76+
# line. Anchored on the parenthesised form so a future change to
77+
# the report's prose doesn't break the parse.
78+
pct=$(grep -oE '\([0-9]+\.[0-9]+%\)' coverage_report.txt | head -1 | tr -d '()%')
79+
if [ -z "$pct" ]; then
80+
echo "::error::Could not parse coverage percentage from coverage_report.txt"
81+
cat coverage_report.txt || true
82+
exit 1
83+
fi
84+
echo "coverage=${pct}" >>"$GITHUB_OUTPUT"
85+
echo "coverage: ${pct}% (audit log: $DOCCANO_FIRED_ROUTES_FILE)"
86+
87+
docker compose down -v --remove-orphans

0 commit comments

Comments
 (0)