Skip to content

Commit b3f8181

Browse files
igerberclaude
andcommitted
Address PR #463 CI R1 (2 P3) — REPORTING parity + PDF reproducibility
CI codex Round 1 verdict was ✅ Looks good. Addressing the two actionable P3 doc findings; the other 4 P3 findings were explicit "no fix required" mitigation confirmations. - P3 REPORTING.md parity: the diagonal-covariance-fallback Note still called the deviation "conservative" at REPORTING.md:324-337, but the new authoritative REGISTRY.md note (in this PR) describes it as a non-paper approximation that is NOT provably conservative (direction of the discrepancy with full Σ_22 depends on sign + magnitude of dropped correlations). Aligned the REPORTING note to match the REGISTRY language, kept the description of the BusinessReport well_powered → moderately_powered downgrade as a practical safeguard (not a proof of conservatism), and cross-referenced the REGISTRY authoritative deviation block. - P3 paper-review reproducibility: the roth-2022-review.md header listed "PDF reviewed: papers/roth-2022.pdf" but the /papers/ directory is gitignored, so the path isn't reproducible from the repo alone. Replaced the path with the published-article citation + DOI + an explicit note that the PDF was reviewed externally and that the /papers/ working directory is gitignored. Added pointers to the jonathandroth.com author page for reproduction. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 3350d0b commit b3f8181

2 files changed

Lines changed: 16 additions & 9 deletions

File tree

docs/methodology/REPORTING.md

Lines changed: 14 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -328,13 +328,20 @@ a library setting.
328328
`DiagnosticReport.pretrends_power` block records
329329
`covariance_source: "diag_fallback_available_full_vcov_unused"` in
330330
that case, and `BusinessReport` downgrades a `well_powered` tier to
331-
`moderately_powered` before rendering prose. This is a known
332-
conservative deviation from the documented "use the full pre-period
333-
covariance" position — it prevents the diagonal approximation from
334-
producing an overly optimistic "well-powered" claim when correlated
335-
pre-period errors could tighten the MDV. The right long-term fix is
336-
to teach `compute_pretrends_power()` to consume `event_study_vcov`
337-
and `event_study_vcov_index`; until that lands this downgrade stays.
331+
`moderately_powered` before rendering prose. This is a documented
332+
deviation from the paper-derived "use the full pre-period covariance"
333+
position. **Not provably conservative**: under Roth (2022)'s NIS
334+
framework and the library's Wald form, the MDV/power objects depend
335+
on the off-diagonals of Σ_22, and the direction of the discrepancy
336+
between full-Σ_22 and diag(ses^2) depends on the sign and magnitude
337+
of the dropped correlations — see the `**Note (deviation from paper
338+
— diagonal pre-period VCV fallback):**` block under `## PreTrendsPower`
339+
in `docs/methodology/REGISTRY.md`. The `well_powered → moderately_powered`
340+
downgrade in BusinessReport reduces the chance of an overly optimistic
341+
claim in practice, but it is not a proof of conservatism. The right
342+
long-term fix is to teach `compute_pretrends_power()` to consume
343+
`event_study_vcov` and `event_study_vcov_index`; until that lands the
344+
downgrade stays.
338345

339346
- **Note:** Unit-translation policy. BusinessReport does not
340347
arithmetically translate log-points to percents or level effects to

docs/methodology/papers/roth-2022-review.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,9 @@
22

33
**Authors:** Jonathan Roth
44
**Citation:** Roth, J. (2022). Pretest with Caution: Event-Study Estimates after Testing for Parallel Trends. *American Economic Review: Insights*, 4(3), 305-322.
5-
**PDF reviewed:** papers/roth-2022.pdf (18 pages, content pages 1-15)
6-
**Review date:** 2026-05-16
75
**DOI:** https://doi.org/10.1257/aeri.20210236
6+
**Source reviewed:** AER:I 4(3), 305-322 (18 pages, content pages 1-15). PDF was reviewed externally and is not committed to the repository (the `/papers/` working directory is gitignored). Reproduce by downloading the published article via the DOI above or from the author's page at https://www.jonathandroth.com/.
7+
**Review date:** 2026-05-16
88

99
---
1010

0 commit comments

Comments
 (0)