Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 29 additions & 1 deletion METHODOLOGY.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,9 +52,37 @@ Three files in this repo:

Detector source and per-finding evidence stay private. The aggregate counts and dispositions are public so the receipt's claims are independently checkable from the ledger alone.

## Methodology vs. policy

The receipt represents the methodology. The CI behaviour represents your team's policy. These are deliberately decoupled.

A check's calibrated severity reflects what the calibration measurement supports. A check that has cleared the bar on one corpus but not two stays at warning severity until cross-corpus confirmation. The receipt's `status` reflects this honestly.

Whether your CI fails on a finding is a separate question, owned by your team. The Action's `fail-on-findings` input lets you opt into hard-stop enforcement. The receipt's severity claim does not move when you do; only the CI behaviour does. Severity belongs to the discipline. Enforcement belongs to you.

## Suppressions and intent

Verify does not infer author intent. Sometimes a finding fires on something the operator has decided is correct — a dev cluster running an edge tag, an internal action they trust. The receipt records that intent only when the operator declares it explicitly.

The mechanism is an in-band comment on the trigger line or the line immediately above it:

```
# verify:ignore <SHAPE-ID> reason:"<text>"
```

A suppression must declare a real shape ID and a non-empty reason. When a suppression is recognized:

- The finding moves out of the primary findings list and into the receipt's "Manifest Intent / Suppressions" block.
- The file path, shape ID, comment line, and verbatim reason are recorded.
- The suppression record is part of the receipt's SHA-256 packet digest.

The packet digest covering suppressions is the load-bearing property here. A suppression cannot be silently removed: doing so changes the digest. A reviewer reading two receipts can tell whether a suppression was added, removed, or modified between commits. The audit chain holds.

Comments that look like suppressions but don't match a fired finding (typo'd shape ID, comment placed where no detector fired) are recorded as suppression warnings rather than silently treated as no-ops. Silent failures are the enemy of auditable systems.

## Reproducing a receipt

The receipt is byte-deterministic. Identical inputs (scan root, source commit, generated-at timestamp, Action bundle version) produce a byte-identical artifact.
The receipt is byte-deterministic. Identical inputs (scan root, source commit, generated-at timestamp, Action bundle version, including any in-band suppression comments) produce a byte-identical artifact.

```
git clone <repo> && cd <repo> && git checkout <commit>
Expand Down
44 changes: 40 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Verify

**Posts a PR change receipt showing what was checked, what was found, and what was not checked. Covers Kubernetes, Dockerfile, and GitHub Actions.**
**Posts a PR change receipt showing what was checked, what was found, and what was not checked. Covers Kubernetes, Dockerfile, GitHub Actions, and one Terraform shape (warning-only).**

The receipt is the product. On every pull request, Verify writes a short artifact that names which checks ran, which fired and where, which ran and were clear, and what was deliberately not checked. Every check that runs carries a published precision number measured on a pinned third-party corpus under a pre-registered rubric.

Expand Down Expand Up @@ -35,9 +35,42 @@ jobs:
- uses: Born14/verify@v1
```

## Optional: fail CI on findings

By default the receipt is informational. To make the workflow fail when there are non-suppressed findings, add `fail-on-findings: true`:

```yaml
- uses: Born14/verify@v1
with:
fail-on-findings: true
```

Suppressed findings (declared with an in-band `# verify:ignore <SHAPE-ID> reason:"..."` comment) do not trigger the failure. They are still recorded in the receipt and covered by the digest.

## Suppressing a finding intentionally

Sometimes a finding fires on something you've decided is correct (a dev cluster running an edge tag, an internal action you trust). Verify lets you declare that intent in-band, on the line where the finding fires:

```yaml
spec:
containers:
# verify:ignore K8S-MISSING-LIMITS-01 reason:"unbounded by design pending sizing review"
- name: api
image: ghcr.io/example/api:1.4.2
```

A suppression must:

- Use the exact syntax `# verify:ignore <SHAPE-ID> reason:"<text>"`
- Be on the same line as the trigger (trailing) or the line immediately above it
- Name a real shape ID (typos are reported as warnings)
- Provide a non-empty reason

Suppressions move the finding out of the primary findings list and into a separate "Manifest Intent / Suppressions" block on the receipt. They are part of the digest, so they cannot be silently removed without changing the receipt's hash. A reviewer reading the receipt sees both the suppression and the reason.

## What gets checked

Seven calibrated checks. Each links to a calibration ledger entry that supports the published precision.
Eight calibrated checks. Each links to a calibration ledger entry that supports the published precision.

| Shape | Surface | Calibrated precision | Risk family |
|---|---|---|---|
Expand All @@ -48,14 +81,17 @@ Seven calibrated checks. Each links to a calibration ledger entry that supports
| `K8S-IMAGE-TAG-LATEST-01` | Kubernetes | 100.00% / 62 | supply-chain drift |
| `GHA-SHA-PIN-01` | GitHub Actions | 75.00% / 20 + 69.44% / 37 | supply-chain drift |
| `DOCKERFILE-BASE-IMAGE-DIGEST-UNPINNED-01` | Dockerfile | 100.00% / 30 + 100.00% / 14 | supply-chain drift |
| `TF-SG-WORLD-OPEN-INGRESS-01` | Terraform | 100.00% / 32 (warning-only, strong-single-corpus) | public exposure |

The Terraform shape is the first calibrated Terraform row. It is narrow but real: one shape, warning-only, single-corpus, measured under the v1.1 same-module Terraform resolver. Not "Terraform solved" and not blocking-tier; future Terraform shapes ship when each one earns its own ledger row.

The supporting calibration ledger lives in this repo at [calibration/](./calibration/): every shape, every corpus, and every attempt that produced the precision numbers above.

## What is not checked

This list ships in the receipt itself on every PR. Excerpted here so it's visible before install:

- Terraform .tf files are not parsed.
- Most Terraform .tf surface area is not parsed; only the security-group ingress shape (`TF-SG-WORLD-OPEN-INGRESS-01`, warning-only) is calibrated under the v1.1 same-module resolver.
- CloudFormation templates are not parsed.
- Helm-templated YAML (`{{ ... }}` expressions) is skipped at detector level.
- Kustomize overlays are not resolved.
Expand All @@ -70,6 +106,6 @@ If you want a different digest, change one of those inputs. If you get a differe

## Status

Free to use. Terraform support is paused on public-corpus availability; the receipt's Not Checked block names that boundary explicitly until a Terraform check calibrates.
Free to use. The first calibrated Terraform shape ships in v1.3 (warning-only, narrow but real); the rest of the Terraform surface remains uncovered, and the receipt's Not Checked block names that boundary explicitly until additional Terraform checks calibrate.

The public calibration ledger lives in this repo at [calibration/](./calibration/). It includes every shape Verify ships, every corpus those shapes were measured against, and every calibration attempt — the ones that promoted and the ones that did not.
6 changes: 5 additions & 1 deletion action.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
name: 'Verify by Born14'
description: 'Posts a PR change receipt showing what was checked, what was found, and what was not checked. Covers Kubernetes, Dockerfile, and GitHub Actions.'
description: 'Posts a PR change receipt showing what was checked, what was found, and what was not checked. Covers Kubernetes, Dockerfile, GitHub Actions, and one Terraform shape (warning-only).'
author: 'Born14'

branding:
Expand All @@ -15,6 +15,10 @@ inputs:
description: 'Directory to scan. Default is the repo root.'
required: false
default: '.'
fail-on-findings:
description: 'When true, the workflow exits non-zero if the receipt has any non-suppressed findings. Suppressions declared in-band with `# verify:ignore <SHAPE-ID> reason:"..."` comments do not trigger failure. Default false; the receipt is informational unless this is set.'
required: false
default: 'false'

outputs:
result:
Expand Down
1 change: 1 addition & 0 deletions dist/action/calibration/attempts.jsonl
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,4 @@
{"attempt_id":"k8s-image-tag-latest-01-v1-iac-argo-cd-v1-2026-04-29","shape_id":"K8S-IMAGE-TAG-LATEST-01","corpus_id":"iac-argo-cd-v1","date":"2026-04-29","detector_version":"v1","rubric_commit":"2a670a4","rubric_binding_commit":"2a670a4","measurement_commit":"d46d426","tp":62,"fp":0,"ambiguous":0,"total_findings":62,"precision":1,"sample_floor_cleared":true,"disposition":"promoted","held_reason":null,"tier_before":"shipped","tier_after":"calibrated","severity_before":"uncalibrated","severity_after":"warning","evidence_path":"calibration/staging/runs/stage-K8S-IMAGE-TAG-LATEST-01-iac-argo-cd-v1-2026-04-30T00-42-46-974Z/classified.jsonl","evidence_public":true,"notes":["First-promotion measurement for K8S-IMAGE-TAG-LATEST-01. Strong-single-corpus path: 62 true positives, 0 false positives, 0 ambiguous on 62-finding clean denominator = 100.00% precision, 0.00% ambiguity. Cleanest IaC ratification of session in TP/false-positive and ambiguity distribution.","FOURTH consecutive operator-configured shape to ratify at 100% on argo-cd: K8S-MISSING-LIMITS-01 (149/0/0), K8S-MISSING-PROBES-01 (41/0/0), K8S-MISSING-SECURITY-CONTEXT-01 (14/0/0), K8S-IMAGE-TAG-LATEST-01 (62/0/0).","Pre-reg a false-positive bucket enumeration empirically validated: anticipated 49 raw matches in resource_customizations/*/health_test.yaml; production-context filter caught most via testdata path; a false-positive bucket regex routing handled remainder. Zero classifier slop.","lessons from a prior cycle applied successfully: pre-reg explicitly noted a prior-cycle lesson and kept false-positive and ambiguity clauses deliberately narrow. Result: zero clause-swallow problem.","Predicate-type novelty handled cleanly: first k8s shape with string-pattern predicate (image tag check). Detector handles digest pinning, registry-with-port edge cases, semantic version tags, no-tag cases. 20/20 fixtures pass on first run.","Determinism check passed: mismatch_count=0.","project promotion policy to warning per the project calibration doctrine.","Files scanned 2288, suppressed by production-context filter 1821, template-skipped 25, parse-errors 0, docs-evaluated 1019, docs-relevant 95.","Companion row: k8s-image-tag-latest-01-v1-iac-datadog-agent-v1-2026-04-29 (held below floor; 1 finding correctly routed a false-positive bucket).","See calibration/K8S-IMAGE-TAG-LATEST-01-v1-closeout-2026-04-29.md for full audit trail.","18th calibrated shape overall. Sixth IaC k8s shape. Second IaC ratification from corpus survey pool (commit 75de434)."],"source_run_id":"stage-K8S-IMAGE-TAG-LATEST-01-iac-argo-cd-v1-2026-04-30T00-42-46-974Z","packet_digest":"1adfbcfefab11d48ff0ff35726dd9c5a1fa6833ee09b47a940eb5a322d31d4ba","ecosystems":["iac-k8s"],"axis_status":"none"}
{"attempt_id":"dockerfile-base-image-digest-unpinned-01-v1-iac-grafana-v1-2026-05-02","shape_id":"DOCKERFILE-BASE-IMAGE-DIGEST-UNPINNED-01","corpus_id":"iac-grafana-v1","date":"2026-05-02","detector_version":"v1","rubric_commit":"<pending>","rubric_temporal_status":"pre-registered","rubric_binding_commit":"<pending>","measurement_commit":"1d7b8c5aa9c7a7f4c6db1dfce898e8e202539eca","corpus_source_sha":"81b2e8450179046c2d4e94b19922cbdd88c8ec61","tp":30,"fp":0,"ambiguous":0,"total_findings":30,"precision":1,"sample_floor_cleared":true,"ambiguity_rate":0,"ambiguity_cap_cleared":true,"disposition":"promoted","promotion_path":"two_corpus_standard","tier_before":"shipped","tier_after":"calibrated","severity_before":"warning","severity_after":"warning","ci_behavior_before":"not-yet-deployed","ci_behavior_after":"warning-only","evidence_path":"calibration/staging/runs/stage-DOCKERFILE-BASE-IMAGE-DIGEST-UNPINNED-01-iac-grafana-v1-2026-05-02T06-19-35-012Z/classified.jsonl","evidence_public":true,"role":"primary","ecosystems":["iac-dockerfile"],"source_run_id":"stage-DOCKERFILE-BASE-IMAGE-DIGEST-UNPINNED-01-iac-grafana-v1-2026-05-02T06-19-35-012Z","packet_digest":"3e331dd544274c8731e9d783c0d22f0e6fd5f86f68b108899403fbbcf7c586e3","detector_scope_counters":{"suppressed_arg_base":7,"suppressed_stage_alias":4,"suppressed_scratch":4,"digest_pinned_safe":0,"files_scanned":30,"files_suppressed_nonproduction":0,"files_suppressed_template":0,"files_skipped_no_runtime":0},"notes":["Eighth IaC calibration cycle. Second Dockerfile-surface shape after CONTAINER-ROOT-01. Selected by survey-iac-tier2-density.ts on 2026-05-02 after Terraform a future release was paused on corpus availability.","Two-corpus standard path. iac-grafana-v1 (this row) cleared blocking band: 30 true positives, 0 false positives, 0 ambiguous on 30-finding clean denominator = 100.00% precision, 0.00% ambiguity. iac-airflow-v1 companion also cleared blocking band (14 true positives, 0 false positives, 0 ambiguous = 100.00%, see companion row).","project promotion policy applies: severity locked at warning regardless of band. Tier shipped -> calibrated. CI behavior changes from not-yet-deployed to warning-only.","Production-context filter: 0 files suppressed in grafana corpus.","Detector-level scope counters (NOT classifier ambiguity per pre-reg S2.6): suppressed_arg_base=7 (build-arg FROMs), suppressed_stage_alias=4 (FROM <prior-AS-alias>), suppressed_scratch=4 (FROM scratch). These are the a non-promotion finding moves pre-registered after moby's 80% ambiguity rate in the survey demonstrated build-args dominate the AMB class.","Determinism check passed: re-run produces byte-identical findings.jsonl.","TPs span ubuntu, golang, nginx, node, mysql, prometheus, mcr.microsoft.com (Azure SQL Edge), apache (jmferrer/apache2-reverse-proxy). All public-registry tagged-but-not-digest-pinned references.","Receipt impact: shape now eligible for inclusion in the change-receipt module's checked_shapes list. Receipt language for Dockerfile changes upgrades from one calibrated shape to two (CONTAINER-ROOT-01 + DOCKERFILE-BASE-IMAGE-DIGEST-UNPINNED-01).","Closeout: calibration/DOCKERFILE-BASE-IMAGE-DIGEST-UNPINNED-01-v1-closeout-2026-05-02.md."]}
{"attempt_id":"dockerfile-base-image-digest-unpinned-01-v1-iac-airflow-v1-2026-05-02","shape_id":"DOCKERFILE-BASE-IMAGE-DIGEST-UNPINNED-01","corpus_id":"iac-airflow-v1","date":"2026-05-02","detector_version":"v1","rubric_commit":"<pending>","rubric_temporal_status":"pre-registered","rubric_binding_commit":"<pending>","measurement_commit":"1d7b8c5aa9c7a7f4c6db1dfce898e8e202539eca","corpus_source_sha":"57955085ecfbd92567ac899406dc4269f966635d","tp":14,"fp":0,"ambiguous":0,"total_findings":14,"precision":1,"sample_floor_cleared":true,"ambiguity_rate":0,"ambiguity_cap_cleared":true,"disposition":"promoted","promotion_path":"two_corpus_standard","tier_before":"shipped","tier_after":"calibrated","severity_before":"warning","severity_after":"warning","ci_behavior_before":"not-yet-deployed","ci_behavior_after":"warning-only","evidence_path":"calibration/staging/runs/stage-DOCKERFILE-BASE-IMAGE-DIGEST-UNPINNED-01-iac-airflow-v1-2026-05-02T06-19-35-012Z/classified.jsonl","evidence_public":true,"role":"companion","ecosystems":["iac-dockerfile"],"source_run_id":"stage-DOCKERFILE-BASE-IMAGE-DIGEST-UNPINNED-01-iac-airflow-v1-2026-05-02T06-19-35-012Z","packet_digest":"6826e6abf5d76bd639c37e092a0fa5772781e5ae749338f0ef7d2d0ae825914d","detector_scope_counters":{"suppressed_arg_base":10,"suppressed_stage_alias":0,"suppressed_scratch":1,"digest_pinned_safe":0,"files_scanned":21,"files_suppressed_nonproduction":0,"files_suppressed_template":0,"files_skipped_no_runtime":1},"notes":["Two-corpus standard path companion row to iac-grafana-v1 primary. iac-airflow-v1 cleared blocking band: 14 true positives, 0 false positives, 0 ambiguous on 14-finding clean denominator = 100.00% precision, 0.00% ambiguity.","Sample floor cleared at 14 (>= 10). The survey predicted lit_fire=15 / amb=10 (40% AMB exactly at cap); actual measurement under v1 detector with detector-level ARG-base suppression returned 14 fires (10 ARG-FROMs scoped out at detector level, NOT counted as findings). One Dockerfile skipped because it lacked any runtime instruction (likely a fragment).","TPs concentrated on apache/airflow tagged versions (apache/airflow:3.3.0 x10, apache/airflow:latest x1) plus public.ecr.aws/lambda/python:3.12, eclipse-temurin:21-jre-ubi9-minimal, centos:7. All public-registry tagged-but-not-digest-pinned references.","Determinism check passed.","project promotion policy applies: severity locked at warning. Tier shipped -> calibrated. CI behavior not-yet-deployed -> warning-only.","Closeout: calibration/DOCKERFILE-BASE-IMAGE-DIGEST-UNPINNED-01-v1-closeout-2026-05-02.md."]}
{"attempt_id":"tf-sg-world-open-ingress-01-v1.1-devops-in-the-cloud-2026-05-06","shape_id":"TF-SG-WORLD-OPEN-INGRESS-01","corpus_id":"devops-in-the-cloud","date":"2026-05-06","detector_version":"v1.1","substrate_version":"v1.1","rubric_commit":"v1.1-pre-reg-2026-05-05","rubric_temporal_status":"pre-registered","rubric_binding_commit":"<pending>","measurement_commit":"5446bd58ec0abd05dda5abc466949119432fcfd0","corpus_source_sha":"b57b4e58d85ebda4912cf0ef32af7288352ea681","tp":32,"fp":0,"ambiguous":16,"total_findings":48,"precision":1,"sample_floor_cleared":true,"ambiguity_rate":0.3333,"ambiguity_cap_cleared":true,"disposition":"promoted","promotion_path":"strong_single_corpus","tier_before":"shipped","tier_after":"calibrated","severity_before":"uncalibrated","severity_after":"warning","ci_behavior_before":"not-yet-deployed","ci_behavior_after":"warning-only","evidence_path":"calibration/staging/runs/stage-TF-SG-WORLD-OPEN-INGRESS-01-devops-in-the-cloud-2026-05-06T04-15-00-000Z/classified.jsonl","evidence_public":true,"role":"primary","ecosystems":["iac-terraform"],"source_run_id":"stage-TF-SG-WORLD-OPEN-INGRESS-01-devops-in-the-cloud-2026-05-06T04-15-00-000Z","packet_digest":"ef428daffc5978883adfdb8e19f0bf73e572a42296e9eda0d9035b62e4737337","bucket_breakdown":{"FP.a":0,"FP.b":0,"FP.c":0,"FP.d":0,"FP.e":0,"FP.f":0,"AMB.a":0,"AMB.b":16,"AMB.c":0,"AMB.d":0,"AMB.e":0,"AMB.f":0,"AMB.g":0,"AMB.h":0,"AMB.i":0},"detector_scope_counters":{"files_evaluated":301,"sites_total":1390,"edges_total":1169,"parse_errors_count":0,"sites_evaluated":144,"findings_count":48},"notes":["First calibrated Terraform shape. v1.1 substrate enabled same-module var/local resolution before detector evaluation; v1 pre-registration was not reused because substrate semantics narrowed an ambiguity bucket/an ambiguity bucket.","Strong-single-corpus promotion path. devops-in-the-cloud cleared the clean-denominator threshold with 32 true positives, 0 false positives, 16 ambiguous on 48 findings: 100.00% clean precision and 33.33% ambiguity, below the 40% cap.","project promotion policy applies: severity is warning even though the clean precision is in the blocking band. CI behavior changes from not-yet-deployed to warning-only.","Duplicate-corpus discipline applied. nextgen-sample-app was excluded after byte-identical Terraform finding fingerprints matched devops-in-the-cloud; the runner now emits duplicate-corpus warnings for identical finding fingerprints.","Determinism check passed: extract -> resolve -> detect produced byte-identical findings across two runs.","Resolver correctness anchor: scripts/iac/terraform/resolver.test.ts passed, and TF-SG fixtures passed under v1.1 substrate.","Closeout: calibration/TF-SG-WORLD-OPEN-INGRESS-01-v1.1-closeout-2026-05-06.md."]}
Loading
Loading