[P] Investigate Codex review-environment capabilities (centralized vs per-repo config)

[cbeaulieu-gt]

Parent epic: #273 · Spec gap discovered after #278 merge.

Context

Spec PR #278 (merged) names AGENTS.md as the mechanism by which the Codex GitHub App receives domain-specific review guidance. Open question raised post-merge: AGENTS.md is per-repo, but the Claude model (pr-review/action.yml's 696-line inline prompt) was effectively centralized — every consumer that added uses: glitchwerks/github-actions/.github/workflows/claude-pr-review.yml@v2 got that prompt's behavior applied to their PRs for free.

If the Codex pivot lands with only a github-actions-local AGENTS.md, consumer repos get whatever Codex defaults to — losing the shared review brain.

Investigation goals

1. Codex environments — the user observed Codex has a notion of cloud "environments" where commands can be run during review (per https://chatgpt.com/codex/cloud/settings/environments, seen during PR docs: codex-pivot spec rev 2 — dual-surface (App + Action), shadow-mode phase #278 review attempt). Investigate:
   • What an environment is, exactly
   • Whether environments are per-repo, per-org, or per-user
   • What commands/scripts run inside (setup, pre-review, etc.)
   • Whether an environment can pull files from a remote source (e.g., curl https://raw.githubusercontent.com/glitchwerks/github-actions/main/AGENTS.md) before review fires
   • Whether environment commands have access to the PR diff / repo context the review will use
2. AGENTS.md resolution order — does Codex look beyond the repo root? E.g., .codex/AGENTS.md, ~/AGENTS.md, env-var-pointed paths, App-config-pointed paths
3. App-level configuration — does the Codex App expose org-wide or App-install-wide configuration that applies a baseline prompt to all reviewed repos?
4. @codex review extra-instructions surface — can a comment-posting workflow inject canonical guidance per-PR (e.g., @codex review using the rules in <fenced block>)? Latency cost of this path?

Deliverable

Structured options report listing every plausible Codex-side mechanism for shared/preloaded review config, with:

• Mechanism name
• Per-repo / per-org / per-App scope
• Authoritative source URL (Codex docs, OpenAI blog, GitHub App marketplace listing) + fetched date
• Whether the option recovers the "central prompt" property of the Claude model
• Effort cost on github-actions side
• Effort cost on consumer side

Why this gates #A

If Codex environments (or any other mechanism) provide a way to preload shared config, #A's scope changes: AGENTS.md authoring becomes one input among several rather than the sole answer. The right approach may be a combination — local AGENTS.md plus an environment-side prep script that fetches additional rules.

Gating

Hard gate on #A (#279). Do not start AGENTS.md authoring until this issue lands with a recommendation.

🤖 Generated by Claude Code on behalf of @cbeaulieu-gt

[cbeaulieu-gt]

Investigation report (devops agent, 2026-05-20)

Findings table

Mechanism	Scope	Authoritative source (fetched 2026-05-20)	Recovers central-prompt property?	github-actions effort	Consumer effort	Confidence
Repo-root AGENTS.md with ## Review guidelines	Per-repo	Code review in GitHub	No — each repo must carry its own copy	Author canonical doc	Copy file into every repo (drift risk)	High
Closest-AGENTS.md walk (root → cwd, concatenated)	Per-repo subtree	AGENTS.md guide	No — still bounded by the repo being reviewed; no remote import syntax documented	n/a	n/a	High
~/.codex/AGENTS.md (global/home)	Per-user (CLI machine)	AGENTS.md guide	No for GitHub App — ~/.codex is the CLI user's home; docs do not state the cloud-review runner mounts a user home with this file. unverified: applies to the App review path	n/a	n/a	Medium (negative)
Codex Cloud environments + setup script	Per-environment (workspace-bound; user/org-tier specifics undocumented)	Cloud environments	unverified: setup scripts have internet access and can install/fetch — could in principle curl a canonical AGENTS.md into the workspace before the agent phase. Docs do not confirm environments apply to the GitHub App @codex review path (they describe ChatGPT-launched cloud tasks). Setup-script exports don't persist to the agent phase, suggesting limited prompt-injection surface.	Host canonical AGENTS.md on a stable URL	One-line env setup script per repo (if applicable)	Low — applicability to App review unverified
Team Config in .codex/ (checked into repo)	Per-repo, intended for team sharing	Config basics via search	No — still per-repo. Holds shared sandbox/approval/skills defaults, not a remote-import primitive	Author template	Copy .codex/ into every repo	Medium
Managed configuration (requirements.toml / managed_config.toml)	Org/tenant (ChatGPT Business/Enterprise)	Managed configuration	No for review prompts — enforces sandbox/approval/MCP allowlists, not review-rule content. Docs explicitly do not address cross-repo review governance	n/a	n/a	High (negative)
Per-PR comment override (@codex review using <rules>)	Per-PR	Code review in GitHub	Partial — could post rules from a workflow that fetches them centrally, but reactive rather than baseline	Author workflow that fetches canonical doc and posts	Add caller workflow	Medium

Recommendation

The centralization property the Claude-side claude-pr-review.yml provided does not have a direct equivalent in the Codex GitHub App. Codex's design assumes review rules live in the repo under review (AGENTS.md / .codex/); managed configuration governs sandbox/security, not prompt content.

The most promising path is a two-layer hybrid:

1. Source of truth: keep the canonical review-rules AGENTS.md on glitchwerks/github-actions.
2. Sync mechanism: a reusable GitHub Actions workflow on this library that consumer repos call on a schedule (or on PR open) to materialize the canonical file into the consumer's repo root as AGENTS.md (or .codex/REVIEW.md if Codex resolves it). Restores centralization at the cost of a sync step per consumer.

Codex Cloud environments is worth one more pass before being ruled out. Next concrete validation step: open a test PR on a throwaway consumer repo with (a) a Cloud environment configured with a setup script that writes a sentinel string into ./AGENTS.md, and (b) @codex review invoked — inspect whether the sentinel appears in Codex's review reasoning. If yes, the environment path is viable and dramatically lower-effort than the sync workflow.

Per-PR comment injection is a viable fallback for ad-hoc rules but not for the baseline review brain.

Sources

• Code review in GitHub – Codex (fetched 2026-05-20)
• Custom instructions with AGENTS.md – Codex (fetched 2026-05-20)
• Cloud environments – Codex web (fetched 2026-05-20)
• Codex code review for GitHub pull requests (fetched 2026-05-20)
• Managed configuration – Codex (fetched 2026-05-20)
• Review – Codex app (fetched 2026-05-20)
• Web – Codex (fetched 2026-05-20)
• Config basics – Codex (via search, 2026-05-20)

🤖 Generated by Claude Code on behalf of @cbeaulieu-gt

[chatgpt-codex-connector]

Summary

• Added a new research document that captures the Codex review-configuration options matrix (mechanism, scope, centralization fit, effort, and confidence), matching the investigation findings from the PR trigger context. docs/superpowers/research/2026-05-20-codex-review-config-options.mdL1-L22
• Documented the recommendation that there is no direct Codex GitHub App equivalent for centralized prompt governance, and proposed the two-layer hybrid approach (canonical source + sync workflow). docs/superpowers/research/2026-05-20-codex-review-config-options.mdL24-L34
• Added a concrete follow-up validation experiment for testing whether Codex Cloud environment setup scripts can influence @codex review behavior, plus a source list with fetch dates. docs/superpowers/research/2026-05-20-codex-review-config-options.mdL35-L55

Testing

• ✅ git status --short
• ✅ git commit -m "docs: add codex review config options investigation report"
• ✅ nl -ba docs/superpowers/research/2026-05-20-codex-review-config-options.md | sed -n '1,220p'

Committed on branch work as:

• 3939dd5 — docs: add codex review config options investigation report

Also completed:

• Created PR metadata via make_pr with title [P] Add Codex review-config options investigation report and a body summarizing rationale and changes.

View task →

[cbeaulieu-gt]

Follow-up: community-signal pass + GitHub-issue verification (2026-05-20)

The prior report (above) rated Codex Cloud environments + setup script as "unverified — applicability to App review unknown" with Low confidence. A direct community signal upgrades this to verified.

Smoking gun: openai/codex#20093 (OPEN)

The reporter explicitly distinguishes paths:

• @codex review works on the same PR / code review path works.
• Direct Codex Cloud task creation against the repo environment works and reaches READY with no diff.
• […] A non-review PR comment such as @codex fix the suggested changes or @codex inspect the current PR context… receives this bot reply instead: "To use Codex here, create an environment for this repo."

Translation: the Cloud environment IS used by the @codex review path — the bug in #20093 is that non-review @codex comments fail to resolve the environment. So:

• ✅ Setup scripts in a Codex Cloud environment run before review and write to the workspace the App sees.
• ✅ Confirms a setup script that does curl https://raw.githubusercontent.com/glitchwerks/github-actions/main/AGENTS.md > ./AGENTS.md would make the canonical AGENTS.md visible to the reviewing agent.
• ⚠️ Cannot rely on environments for the fix verb (@codex address that feedback) — that's the broken path in #20093. We use openai/codex-action@v1.8 for the fix-side workflows per spec §5.2–§5.4 anyway, so this doesn't break the plan.

Other findings

Source	Useful?	Notes
openai/codex#3694 (CLOSED)	Low	Generic Script exited with code 1 on review. Closed Jan 2026 as old bug without root-cause comment. Intermittent failure mode worth knowing for #O.
Ruvnet gist	Low	SPARC/Mastra setup script for ChatGPT-launched cloud tasks, not App review. Pattern exists in the wild but not directly portable.
P0/P1 categorization	High	Codex's review surface is bounded to P0/P1 findings. AGENTS.md customization vocabulary follows the pattern ## Review guidelines → "Flag X as P0", "Flag Y as P1". This is the schema to use when porting our 696-line Claude prompt.

Updated recommendation (supersedes prior)

Mechanism	Status post-#20093	Effort
Cloud environment + setup script that fetches canonical AGENTS.md	✅ Verified viable	Author canonical AGENTS.md; document one-line env setup-script template for consumers
Sync-workflow hybrid	Fallback	Higher consumer effort; falls back if env path breaks
Per-PR rule injection	Last resort	Only for ad-hoc additions

Reshape of #A and new sub-issue proposal

Original #A (#279) — "Author top-level AGENTS.md, verify on throwaway PR" — needs to expand to:

1. Author canonical AGENTS.md on glitchwerks/github-actions (P0/P1 vocabulary, ported from pr-review/action.yml's 696-line prompt)
2. Document the consumer-side env-setup-script pattern in docs/consumer-onboarding.md:

   # Example env setup script (paste into Codex Cloud → Settings → Environments)
   curl -fsSL https://raw.githubusercontent.com/glitchwerks/github-actions/v3/AGENTS.md > ./AGENTS.md

3. Verify end-to-end: setup script fetches our AGENTS.md → App reviews a test PR → Codex's review reasoning references guidance from the fetched file

Open question for #20093 risk

The non-review @codex comment path is broken per #20093. This is mostly orthogonal to the cutover plan (fix/lint/ci-failure go through openai/codex-action@v1.8, not @codex comments), but we should confirm none of our intended consumer-side workflows depend on @codex address/@codex fix comment triggers. Filing as a follow-up note rather than a blocking gate — the surfaces we're building use the action, not comment-triggered tasks.

Sources

• openai/codex#20093 — non-review comments fail; review path works (fetched 2026-05-20)
• openai/codex#3694 — review Script exited with code 1 failure mode (fetched 2026-05-20)
• Code review in GitHub – Codex (fetched 2026-05-20)
• Cloud environments – Codex web (fetched 2026-05-20)
• Codex Code Review & Actor – GitHub Marketplace (fetched 2026-05-20)
• Ruvnet gist (fetched 2026-05-20) — not directly portable
• Build Code Review with the Codex SDK – cookbook (fetched 2026-05-20)

🤖 Generated by Claude Code on behalf of @cbeaulieu-gt

[chatgpt-codex-connector]

Summary

• Updated the Codex evaluation research document to record the 2026-05-20 follow-up verification from openai/codex#20093, explicitly upgrading Cloud environment applicability for @codex review to verified and documenting the remaining non-review comment-path caveat. docs/superpowers/research/2026-05-20-codex-evaluation.mdL79-L87
• Added an explicit implication in the same research doc that a setup script can fetch a canonical AGENTS.md into the workspace before review (including the concrete curl -fsSL ... > ./AGENTS.md pattern). docs/superpowers/research/2026-05-20-codex-evaluation.mdL84-L86
• Added Step 8 to consumer onboarding with the recommended Codex Cloud environment setup-script template for centralized AGENTS hydration, plus rationale and a limitation note steering fix-side flows to openai/codex-action@v1.8. docs/consumer-onboarding.mdL233-L251

Testing

• ✅ git status --short
• ✅ git diff -- docs/superpowers/research/2026-05-20-codex-evaluation.md docs/consumer-onboarding.md | sed -n '1,240p'
• ✅ nl -ba docs/consumer-onboarding.md | sed -n '220,280p'
• ✅ nl -ba docs/superpowers/research/2026-05-20-codex-evaluation.md | sed -n '72,104p'
• ✅ git add docs/consumer-onboarding.md docs/superpowers/research/2026-05-20-codex-evaluation.md && git commit -m "docs: update codex environment verification and onboarding script"

Committed on branch work:

• 13a5806 — docs: update codex environment verification and onboarding script

Also completed:

• Created PR metadata via make_pr with title [P] Update Codex environment verification follow-up and onboarding guidance.

View task →

[cbeaulieu-gt]

Correction (2026-05-20): walk back overstatement in previous comment

The previous comment described the Cloud-environment + setup-script path as "✅ Verified viable" with the "Smoking gun" framing. That was overstated. The actual evidence supports only "plausible, with supporting third-party signal" — no empirical test was run.

What's actually known vs. assumed

Claim	Evidence	Strength
@codex review consults a Codex Cloud environment when one is configured	openai/codex#20093 reporter's words: "@codex review works on the same PR / code review path works" + "Direct Cloud task creation against the repo environment works and reaches READY"	Strong — single first-hand reporter, three other +1 confirmations, no OpenAI-staff refutation
The environment's setup script runs before review, and the workspace it writes to is the same one Codex reads during review	Inferred — not stated verbatim in any fetched source. "Cloud environment + setup script" is the documented unit, and #20093 confirms the environment is used; the chain to "setup script writes a file → Codex sees it during review" is not in evidence.	Moderate (inference)
A curl in the setup script can fetch a remote AGENTS.md and Codex will follow its content during review	Not tested. Not stated in any source.	Speculative

Updated mechanism table

Mechanism	Status (corrected)	Effort
Cloud environment + setup script that fetches canonical AGENTS.md	🟡 Plausible, with third-party supporting evidence — needs empirical validation before commitment	Author canonical AGENTS.md; design env setup-script template; run validation experiment
Sync-workflow hybrid	Fallback	Higher consumer effort; commits to repo-resident files instead of env-side fetch
Per-PR rule injection	Last resort	Only for ad-hoc additions

Required next step before committing to the env-script path

Run the validation experiment described above:

1. Create a throwaway test repo with the Codex GitHub App installed
2. Configure a Codex Cloud environment for that repo with a setup script that writes a sentinel string into ./AGENTS.md (e.g., ## Review guidelines\nFlag any function named "sentinel_canary_xyz" as P0)
3. Open a PR adding code that contains that function
4. Observe whether Codex's @codex review flags the sentinel
5. Outcomes:
   • Codex flags it → setup-script + AGENTS.md chain works → env-script path commits as primary
   • Codex does not flag it → fall back to sync-workflow hybrid (or further investigation into why the script didn't take)

This experiment is the new prerequisite for #A. Filing as a follow-up if confirmed by user.

🤖 Generated by Claude Code on behalf of @cbeaulieu-gt

[chatgpt-codex-connector]

Summary

• Added a dated correction note that explicitly walks back overstatement and reclassifies the Cloud-environment + setup-script claim as plausible inference pending empirical validation, not verified fact. docs/superpowers/research/2026-05-20-codex-evaluation.mdL80
• Added a corresponding failure-mode entry documenting the inference risk around environment/app coupling and AGENTS.md visibility, with openai/codex#20093 cited as supporting (but non-conclusive) evidence. docs/superpowers/research/2026-05-20-codex-evaluation.mdL537
• Committed the change on branch work as a7361d4 with message: docs: correct evidence strength for codex env-script review path. docs/superpowers/research/2026-05-20-codex-evaluation.mdL80-L83

Testing

• ✅ git commit -m "docs: correct evidence strength for codex env-script review path"
• ✅ nl -ba docs/superpowers/research/2026-05-20-codex-evaluation.md | sed -n '72,96p'
• ✅ nl -ba docs/superpowers/research/2026-05-20-codex-evaluation.md | sed -n '528,542p'
• ✅ git diff -- docs/superpowers/research/2026-05-20-codex-evaluation.md | sed -n '1,220p'

Also completed: created PR metadata via make_pr with title [P] Correct Codex environment-evidence wording in research report.

View task →

[cbeaulieu-gt]

Deferred pending exploration

Codex evaluation has been pivoted to a scratch repo: glitchwerks/github-actions-codex. See tracking issue #297.

The research in this issue is durable (mechanism comparison, AGENTS.md schema, P0/P1 vocabulary, the openai/codex#20093 finding about env behavior on review). It will inform the scratch-repo exploration but is no longer load-bearing for any near-term work in this repo.

Leaving open as a record. Will revisit when (and if) the exploration returns a positive Codex-as-next-platform decision and Phase 1 sub-issues in this repo resume.

Refs #297.

🤖 Generated by Claude Code on behalf of @cbeaulieu-gt