ci coverage: fail PRs on per-package coverage decrease or overall floor breach

.github/workflows/cli-ci.yml

@@ -6,6 +6,7 @@ on:
       - "cli/**"
       - "!cli/azd/extensions/**"
       - ".github/workflows/cli-ci.yml"
+      - "eng/scripts/Get-CoverageDiff*.ps1"
     branches: [main]
 
 # If two events are triggered within a short time in the same PR, cancel the run of the oldest event
@@ -44,3 +45,34 @@ jobs:
 
   bicep-lint:
     uses: ./.github/workflows/lint-bicep.yml
+
+  coverage-script-tests:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+    steps:
+      - uses: actions/checkout@v6
+      - name: Run Pester tests for coverage scripts
+        shell: pwsh
+        run: |
+          # Pin Pester to a specific signed release for deterministic, supply-chain-safe installs.
+          Install-Module -Name Pester -RequiredVersion 5.7.1 -Force -Scope CurrentUser
+          Import-Module Pester -RequiredVersion 5.7.1
+          $config = New-PesterConfiguration
+          $config.Run.Path = './eng/scripts/Get-CoverageDiff.Tests.ps1'
+          $config.Run.Exit = $true
+          $config.Output.Verbosity = 'Detailed'
+          Invoke-Pester -Configuration $config
+
+  magefile-tests:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+    steps:
+      - uses: actions/checkout@v6
+      - uses: actions/setup-go@v6
+        with:
+          go-version-file: cli/azd/go.mod
+      - name: Run mage helper tests (resolveCoverageFile, resolveBaselineFile, resolveChangedFilesForDiff)
+        working-directory: cli/azd
+        run: go test -tags mage -run '^TestResolve' -v .

cli/azd/.vscode/cspell.yaml

@@ -401,6 +401,16 @@ overrides:
     words:
       - covdata
       - GOWORK
+      - textfmt
+      - logissue
+      - runbook
+      - AMRD
+  - filename: magefile.go
+    words:
+      - textfmt
+      - covcounters
+      - covmeta
+      - AMRD
   - filename: test/eval/README.md
     words:
       - Waza

cli/azd/AGENTS.md

@@ -78,6 +78,7 @@ When writing tests, prefer table-driven tests. Use testify/mock for mocking.
 Additional mage targets:
 
 - `mage record` — re-record functional test cassettes against a live Azure subscription. Accepts an optional `-filter=TestName` flag to re-record specific tests. Typically only core maintainers need to run this; external contributors can rely on playback mode (the default) which requires no Azure access. Requires `azd auth login` and a configured test subscription (see `docs/recording-functional-tests-guide.md`).
+- `mage coverage:pr` — preview the CI PR coverage gate locally before pushing. Resolves PR-touched `.go` files via `git merge-base origin/main HEAD` for the per-package summary, runs the diff against the latest `main` baseline, and fails (exit 2) on **either** breach type: any PR-touched package drops more than 0.5 pp, or overall coverage falls below 69% (defaults match CI; override via `COVERAGE_MAX_PACKAGE_DECREASE`, `COVERAGE_MIN_OVERALL`). See `docs/code-coverage-guide.md` for details.
 
 ```bash
 gofmt -s -w .

cli/azd/docs/code-coverage-guide.md

@@ -39,8 +39,10 @@ The CI pipeline collects coverage in several stages:
 6. **Filter**: `Filter-GeneratedCoverage.ps1` removes auto-generated files
    (e.g., `*.pb.go`) so coverage reflects only hand-written code.
 
-7. **Threshold**: `Test-CodeCoverageThreshold.ps1` enforces the minimum
-   coverage gate, failing the build if coverage drops below the threshold.
+7. **PR gate** (PR builds only): `Get-CoverageDiff.ps1` compares the merged
+   profile against the latest successful `main` baseline and fails the build
+   on either a per-package decrease > 0.5 pp or overall coverage below the
+   floor. Release/main builds skip this step and only publish artifacts.
 
 ## Developer Modes
 
@@ -159,14 +161,22 @@ additional code generators are introduced.
 
 ## CI Gate
 
-The CI pipeline enforces a minimum coverage threshold using
-`Test-CodeCoverageThreshold.ps1` in the `release-cli.yml` pipeline.
-
-- **Current threshold**: Check the pipeline definition for the latest value.
-- **Ratchet policy**: The threshold is periodically raised as coverage improves.
-  PRs that reduce coverage below the threshold will fail the coverage gate.
-- **Enforcement**: The threshold script parses `go tool cover -func` output and
-  exits non-zero if the total statement coverage is below the minimum.
+The CI pipeline enforces a two-gate coverage check on **PR builds only**
+using `Get-CoverageDiff.ps1` (invoked from
+`eng/pipelines/templates/stages/code-coverage-upload.yml`):
+
+- **Per-package gate**: any PR-touched package that drops more than
+  `MaxPackageDecrease` percentage points (default **0.5 pp**) versus the
+  latest successful `main` baseline fails the build.
+- **Overall floor gate**: if the PR's overall coverage falls below
+  `MinOverallCoverage` (default **69%**), the build fails.
+- **Failure mode**: the script emits `##vso[task.logissue type=error]`
+  for each breached gate and exits with code `2`, which fails the ADO job.
+- **Scope**: release, scheduled, and `main` builds **do not** enforce
+  these gates — they only publish coverage artifacts. A coverage dip on
+  `main` will surface on the next PR rather than block a release.
+- **Ratchet policy**: see the *Adjusting the absolute floor* runbook
+  below.
 
 ## Scripts Reference
 
@@ -175,7 +185,8 @@ The CI pipeline enforces a minimum coverage threshold using
 | `Get-LocalCoverageReport.ps1` | `eng/scripts/` | Developer-facing: runs coverage locally in any of the 4 modes |
 | `Get-CICoverageReport.ps1` | `eng/scripts/` | Downloads combined coverage from Azure DevOps CI builds |
 | `Filter-GeneratedCoverage.ps1` | `eng/scripts/` | Strips auto-generated files (`.pb.go`) from coverage profiles |
-| `Test-CodeCoverageThreshold.ps1` | `eng/scripts/` | Enforces minimum coverage gate; used by CI and `-MinCoverage` |
+| `Get-CoverageDiff.ps1` | `eng/scripts/` | PR coverage gate: two-gate check (per-package decrease + overall floor) used by CI and `mage coverage:pr` |
+| `Test-CodeCoverageThreshold.ps1` | `eng/scripts/` | Local minimum-coverage helper used by `Get-LocalCoverageReport.ps1 -MinCoverage` (no longer wired into CI) |
 | `Convert-GoCoverageToCobertura.ps1` | `eng/scripts/` | Converts Go coverage to Cobertura XML for ADO reporting (CI only) |
 | `ci-build.ps1` | `eng/scripts/` | CI: builds azd binary with `-cover` instrumentation |
 | `ci-test.ps1` | `eng/scripts/` | CI: runs unit and integration tests with coverage collection |
@@ -192,8 +203,9 @@ All modes are also available as `mage` targets (from `cli/azd/`):
 | `mage coverage:ci` | CI baseline report | `az login` |
 | `mage coverage:html` | HTML report (unit only by default) | Go 1.26 |
 | `mage coverage:check` | Enforce 50% threshold (unit only; CI gate is 55% combined) | Go 1.26 |
-| `mage coverage:diff` | Compare current branch coverage vs main baseline | Go 1.26 |
-| `mage coverage:pr` | Diff + post results as a PR comment | Go 1.26, `gh` CLI |
+| `mage coverage:diff` | Compare current branch coverage vs main baseline (advisory; honors `COVERAGE_MAX_PACKAGE_DECREASE` / `COVERAGE_MIN_OVERALL` / `COVERAGE_FAIL_ON_DECREASE`) | Go 1.26 |
+| `mage coverage:pr` | Preview the CI PR coverage gate locally (fail-loud on either: per-package regression > 0.5 pp, or overall < 69%) | Go 1.26 |
+| `mage coverage:report` | Merge raw covdata input directories into a single `cover.out` (used by CI; honors `COVERAGE_REPORT_*` env vars) | Go 1.26 |
 
 Environment variables for optional overrides:
 
@@ -203,6 +215,175 @@ Environment variables for optional overrides:
 | `COVERAGE_BUILD_ID` | `hybrid`, `ci` | Target a specific ADO build ID |
 | `COVERAGE_MODE` | `html` | Set to `full` or `hybrid` (default: `unit`) |
 | `COVERAGE_MIN` | `check` | Override threshold (default: `55`) |
+| `COVERAGE_MAX_PACKAGE_DECREASE` | `diff`, `pr` | Maximum tolerated per-package coverage drop in percentage points (defaults come from `Get-CoverageDiff.ps1`, currently `0.5`; PR-touched packages only when changed-files can be resolved). Set to `-1` to disable the per-package gate (the floor gate stays active unless `COVERAGE_MIN_OVERALL` is also set to `-1`). |
+| `COVERAGE_MIN_OVERALL` | `diff`, `pr` | Absolute floor for overall coverage in percent (defaults come from `Get-CoverageDiff.ps1`, currently `69`). Set to `-1` to disable the floor gate. |
+| `COVERAGE_FAIL_ON_DECREASE` | `diff` | Set to `1` / `true` to exit `2` when EITHER gate is breached (`pr` always fails loud). Any other non-zero exit indicates a script/infra error, not a gate breach. **Note:** setting `COVERAGE_MAX_PACKAGE_DECREASE` alone does NOT enable fail-loud mode for `mage coverage:diff` — you must also set `COVERAGE_FAIL_ON_DECREASE=1` (or use `mage coverage:pr`, which always fails loud). |
+| `COVERAGE_BASELINE` | `diff`, `pr` | Path to baseline coverage profile (default: `cover-ci-combined.out` or download from CI) |
+| `COVERAGE_CURRENT` | `diff`, `pr` | Path to current coverage profile (default: `cover-local.out`) |
+| `COVERAGE_REPORT_UNIT_INPUTS` | `report` | Comma-separated list of unit-test covdata input directories. |
+| `COVERAGE_REPORT_INT_INPUTS` | `report` | Comma-separated list of integration-test covdata input directories (optional). |
+| `COVERAGE_REPORT_OUTPUT` | `report` | Output `cover.out` path (textfmt). |
+| `COVERAGE_REPORT_MERGED_DIR` | `report` | Optional intermediate merged covdata directory. Created if absent. |
+
+## PR Coverage Check (Fail-Loud)
+
+PRs run a **two-gate** coverage check as part of the
+`code-coverage-upload.yml` Azure DevOps stage. After unit + integration
+coverage is merged via `mage coverage:report`, the pipeline:
+
+1. Resolves the list of `.go` files touched by the PR via
+   `git diff --name-only --no-renames --diff-filter=AMRD origin/<targetBranch>...HEAD`,
+   so per-package results are scoped to the packages this PR touches.
+2. Runs `eng/scripts/Get-CoverageDiff.ps1` against the merged baseline
+   from the latest successful build of the PR target branch and the PR's `cover.out`.
+3. Prints a per-package report (regressions first), the overall delta,
+   and the configured tolerances.
+4. **Fails the build (`exit 2`) when EITHER of the following is true:**
+   - **Per-package decrease**: any single PR-touched package drops by
+     more than `MaxPackageDecrease` pp (default **0.5 pp**).
+   - **Absolute floor**: overall coverage falls below
+     `MinOverallCoverage` percent (default **69%**).
+5. Surfaces every breach via `##vso[task.logissue type=error]` so each
+   one shows up in the PR check summary.
+
+Per-package results outside the PR-touched set are advisory; they appear
+in the report but do not gate the build. There is intentionally **no PR
+comment**; the build log is the source of truth.
+
+### Reproducing the gate locally
+
+```powershell
+# 1. Build the unit-only profile for your branch
+mage coverage:unit
+
+# 2. Run the same gate CI runs
+mage coverage:pr
+```
+
+`mage coverage:pr` runs `git fetch --no-tags --depth=200 origin main` (best-effort),
+resolves changed files via `git merge-base origin/main HEAD` for the
+per-package report, applies the default 0.5 pp per-package tolerance and
+69% absolute floor, and exits with code `2` when either is breached (any
+other non-zero exit indicates a script/infra error). On `main`,
+in detached-HEAD state, or when git resolution fails, the target returns
+an error rather than silently passing (the "preview" guarantee depends on
+running against the same inputs CI uses). For an advisory run on `main`,
+use `mage coverage:diff` instead.
+
+### Configuring the tolerance
+
+Override per run:
+
+```powershell
+$env:COVERAGE_MAX_PACKAGE_DECREASE = "1.0"; mage coverage:pr
+```
+
+Or use the advisory `coverage:diff` target with explicit opt-in:
+
+```powershell
+$env:COVERAGE_MAX_PACKAGE_DECREASE = "0.5"
+$env:COVERAGE_MIN_OVERALL = "69"
+$env:COVERAGE_FAIL_ON_DECREASE = "1"
+mage coverage:diff
+```
+
+### Adjusting the absolute floor (`MinOverallCoverage`)
+
+The PR pipeline fails when overall coverage falls below
+`MinOverallCoverage` (default **69%**). The floor is calibrated just below
+the observed main overall coverage so it ratchets quality up while leaving
+a small safety margin for normal churn. The release / `main` / scheduled
+pipelines do not enforce the floor — `CodeCoverage_Upload` runs there only
+to publish coverage artifacts. When a wave of refactors or generated-code
+changes shifts overall coverage below the floor, follow this runbook so
+PRs don't get jammed:
+
+1. **Confirm the dip is real** — pull the latest combined profile and read
+   the overall number:
+
+   ```powershell
+   mage coverage:ci
+   go tool cover "-func=cover-ci-combined.out" | Select-String "^total:"
+   ```
+
+   If the dip is genuine (not a flaky platform leg producing artifact gaps),
+   continue.
+
+2. **Lower the floor temporarily** in
+   `eng/pipelines/templates/stages/code-coverage-upload.yml` — find the
+   `MinOverallCoverage` parameter and set `default:` to **1pp below** the new
+   observed overall (e.g. observed 62.5% → set 61). This unblocks `main`.
+
+3. **File a coverage-debt issue** capturing: the package(s) responsible for
+   the regression, the floor delta you applied, and a target date to ratchet
+   back. Without this step the floor silently stays loose forever.
+
+4. **Ratchet the floor back up** once the responsible package(s) regain
+   coverage. Bump `default:` to **1pp below the new observed overall**.
+   Avoid moving the floor in increments larger than 2pp at a time so that a
+   single flaky monthly measurement can't lock in an artificially-high
+   floor.
+
+> ⚠️ **Branch-protection note (one-time setup, repo admin):** the gate's
+> exit code is only enforced on merges if the `CodeCoverage_Upload` stage
+> is configured as a **required status check** on `main` in the repo's
+> branch-protection rules. Without this, a PR can merge while the coverage
+> check is red.
+>
+> A repo admin can confirm or add the rule via the GitHub UI
+> (`Settings → Branches → Branch protection rules → main → Require status
+> checks to pass before merging`), selecting the
+> `azure-dev - ci - CodeCoverage_Upload` check.
+>
+> Equivalent `gh` CLI command (run by an admin once after the first
+> successful build of this PR):
+>
+> ```bash
+> gh api -X PATCH repos/Azure/azure-dev/branches/main/protection \
+>     -f 'required_status_checks[strict]=true' \
+>     -f 'required_status_checks[contexts][]=azure-dev - ci - CodeCoverage_Upload'
+> ```
+>
+> The exact context name comes from the GitHub Checks tab on a PR build —
+> verify the string before applying. The check appears only after the
+> stage has run on at least one PR, so seed it by opening a draft PR
+> first.
+
+### Worked example
+
+Suppose this PR touches `pkg/auth` and drops its coverage from 72.0% → 48.0% (a
+24.0 pp drop, well past the 0.5 pp tolerance). Overall coverage stays at 70.0%
+(comfortably above the 69% floor — passes). The CI step prints:
+
+```
+============================================================
+Coverage Report
+============================================================
+Baseline: baseline
+Overall: 70.4% -> 70.0% (-0.4%)
+  Tolerance: -0.5 pp per package before failing the gate
+  Floor: overall coverage must stay >= 69.0%
+PR-touched packages (2 packages):
+  pkg/auth                  72.0% ->  48.0%  ( -24.0%)  regress  (1 file touched)
+  pkg/project               81.0% ->  82.0%  (  +1.0%)  improved (2 files touched)
+============================================================
+RESULT: FAIL
+============================================================
+Breached gate(s):
+  - 1 package(s) dropped more than 0.5 pp:
+      pkg/auth: 72.0% -> 48.0% (-24.0 pp)
+============================================================
+```
+
+…then emits:
+
+```
+##vso[task.logissue type=error]Package pkg/auth dropped 24.0 pp (max allowed: -0.5 pp).
+```
+
+…and exits 2. The PR check summary shows the error, and the build fails. If
+overall coverage had also fallen below 69.0%, a second `##vso[task.logissue]`
+line would name the floor breach (both gates report independently).
 
 ## Troubleshooting
 

cli/azd/go.mod

@@ -53,7 +53,7 @@ require (
 	github.com/invopop/jsonschema v0.13.0
 	github.com/jmespath-community/go-jmespath v1.1.1
 	github.com/joho/godotenv v1.5.1
-	github.com/magefile/mage v1.16.0
+	github.com/magefile/mage v1.17.2
 	github.com/mark3labs/mcp-go v0.41.1
 	github.com/mattn/go-colorable v0.1.14
 	github.com/mattn/go-isatty v0.0.20
@@ -81,6 +81,7 @@ require (
 	go.uber.org/atomic v1.11.0
 	go.uber.org/multierr v1.11.0
 	go.yaml.in/yaml/v3 v3.0.4
+	golang.org/x/sync v0.20.0
 	golang.org/x/sys v0.42.0
 	golang.org/x/term v0.41.0
 	golang.org/x/time v0.9.0
@@ -148,7 +149,6 @@ require (
 	golang.org/x/crypto v0.49.0 // indirect
 	golang.org/x/exp v0.0.0-20250911091902-df9299821621 // indirect
 	golang.org/x/net v0.52.0 // indirect
-	golang.org/x/sync v0.20.0 // indirect
 	golang.org/x/text v0.35.0 // indirect
 	google.golang.org/genproto/googleapis/api v0.0.0-20260401024825-9d38bb4040a9 // indirect
 )

cli/azd/go.sum

@@ -211,8 +211,8 @@ github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0
 github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
 github.com/lucasb-eyer/go-colorful v1.3.0 h1:2/yBRLdWBZKrf7gB40FoiKfAWYQ0lqNcbuQwVHXptag=
 github.com/lucasb-eyer/go-colorful v1.3.0/go.mod h1:R4dSotOR9KMtayYi1e77YzuveK+i7ruzyGqttikkLy0=
-github.com/magefile/mage v1.16.0 h1:2naaPmNwrMicCdLBCRDw288hcyClO9lmnm6FMpXyJ5I=
-github.com/magefile/mage v1.16.0/go.mod h1:z5UZb/iS3GoOSn0JgWuiw7dxlurVYTu+/jHXqQg881A=
+github.com/magefile/mage v1.17.2 h1:fyXVu1eadI8Ap1HCCNgEhJ5McIWiYhLR8uol64ZZc40=
+github.com/magefile/mage v1.17.2/go.mod h1:Yj51kqllmsgFpvvSzgrZPK9WtluG3kUhFaBUVLo4feA=
 github.com/mailru/easyjson v0.9.1 h1:LbtsOm5WAswyWbvTEOqhypdPeZzHavpZx96/n553mR8=
 github.com/mailru/easyjson v0.9.1/go.mod h1:1+xMtQp2MRNVL/V1bOzuP3aP8VNwRW55fQUto+XFtTU=
 github.com/mark3labs/mcp-go v0.41.1 h1:w78eWfiQam2i8ICL7AL0WFiq7KHNJQ6UB53ZVtH4KGA=