Generic autosolve github workflow for automated issue resolution

[fantapop]

No description provided.

[Copilot]

Pull request overview

Adds new reusable GitHub Actions/workflows to support automated “autosolve” (assess + implement) flows and changelog-driven release tagging, along with a lightweight bash test harness and CI wiring for the repo.

Changes:

• Introduce autotag-from-changelog composite action + script + tests to create/push tags based on CHANGELOG.md.
• Add autosolve composite actions (assess, implement), shared bash utilities, prompts, and reusable workflows (Jira + GitHub Issue).
• Add bash test framework (test.sh, test_helpers.sh) plus CI workflow to run tests on PRs.

Reviewed changes

Copilot reviewed 28 out of 28 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
test_helpers.sh	Adds shared bash assertions/helpers for repository test scripts.
test.sh	Adds a simple test runner that discovers and executes *_test.sh files.
autotag-from-changelog/auto-tag-release.sh	Implements tagging logic based on CHANGELOG.md state.
autotag-from-changelog/auto-tag-release_test.sh	Tests tagging behavior using temporary git repos.
autotag-from-changelog/action.yml	Composite action wrapper for the changelog autotag script.
autosolve/scripts/shared.sh	Shared autosolve functions (validation, prompt building, result parsing, CLI install).
autosolve/scripts/shared_test.sh	Unit tests for shared autosolve functions.
autosolve/scripts/assess.sh	Runs read-only Claude assessment and extracts structured outputs.
autosolve/scripts/assess_test.sh	Tests assess output formatting/extraction behavior.
autosolve/scripts/implement.sh	Runs Claude implementation, security validation, push+PR creation, and output plumbing.
autosolve/scripts/implement_test.sh	Tests security_check behavior in a temporary git repo.
autosolve/scripts/jira.sh	Jira prompt building, commenting, transitions, and final status helpers.
autosolve/scripts/jira_test.sh	Tests non-HTTP Jira helper functions.
autosolve/run_step.sh	Entry-point wrapper to run autosolve script functions from the workspace CWD.
autosolve/prompts/security-preamble.md	System/security preamble injected into prompts.
autosolve/prompts/assessment-footer.md	Standardizes assessment output markers.
autosolve/prompts/implementation-footer.md	Standardizes implementation output markers and instructions.
autosolve/assess/action.yml	Composite action wiring for assess flow.
autosolve/implement/action.yml	Composite action wiring for implement flow (incl. security check and PR creation).
actions_helpers.sh	Adds common logging + GitHub Actions output helpers.
actions_helpers_test.sh	Tests for actions_helpers.sh helpers.
README.md	Documents new actions/workflows and local development/testing.
CLAUDE.md	Adds repo conventions and guidance for Claude-driven workflows and testing.
CHANGELOG.md	Adds entries describing the new actions/workflows.
.shellcheckrc	Configures shellcheck behavior for repo sourcing patterns.
.github/workflows/test.yml	Adds CI job to run ./test.sh on PRs.
.github/workflows/jira-autosolve.yml	Adds reusable Jira autosolve workflow that composes assess+implement.
.github/workflows/github-issue-autosolve.yml	Adds reusable GitHub Issue autosolve workflow (assess+implement + commenting/label mgmt).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

[fantapop]

Done — see reply above, all three checks now use grep -F for literal matching.

[fantapop]

Feedback from testing in ccloud-private-automation-testing

While building a test workflow that uses the composite actions directly (not the reusable workflows), I noticed the README's auth documentation is a bit thin for direct action usage.

The reusable workflows (github-issue-autosolve.yml, jira-autosolve.yml) handle auth setup internally — they accept auth_mode, vertex_project_id, etc. as inputs and set the right env vars on each step. But when using autosolve/assess and autosolve/implement directly, the caller needs to know to:

1. Run google-github-actions/auth@v3 (or equivalent) themselves
2. Set the env vars (CLAUDE_CODE_USE_VERTEX, ANTHROPIC_VERTEX_PROJECT_ID, CLOUD_ML_REGION) on each action step

This works fine once you read github-issue-autosolve.yml as a reference, but the README's "Required authentication" section only mentions the reusable workflow inputs (auth_mode: vertex). It would be helpful to add a note or example showing the env vars needed for direct composite action usage. Something like:

# Direct action usage with Vertex AI
- uses: google-github-actions/auth@v3
  with:
    project_id: my-project
    service_account: my-sa@my-project.iam.gserviceaccount.com
    workload_identity_provider: projects/.../providers/...

- uses: cockroachdb/actions/autosolve/assess@v1
  env:
    CLAUDE_CODE_USE_VERTEX: "1"
    ANTHROPIC_VERTEX_PROJECT_ID: my-project
    CLOUD_ML_REGION: us-east5
  with:
    prompt: "Fix the bug"

[fantapop]

Bug: status reports SUCCESS when push_and_pr fails

When the implementation succeeds and security check passes, but the push_and_pr step fails (e.g., fork repo not accessible), the final status is still reported as SUCCESS.

From a test run log:

=== Final Result ===
Assessment: PROCEED
Implementation status: SUCCESS
PR URL: 
Branch: 

The implementation step succeeded (Claude created the file), security check passed, but push_and_pr failed with exit code 128 (repo not found). Despite this, set_implement_outputs reported status=SUCCESS because it only checks IMPL_RESULT and SECURITY_CONCLUSION:

# implement.sh:set_implement_outputs()
if [ "$impl_result" = "SUCCESS" ] && [ "$security_conclusion" != "failure" ]; then
    status="SUCCESS"

It doesn't check whether push_and_pr actually succeeded. The PR_URL and BRANCH_NAME being empty are clues, but the status itself is misleading.

Suggested fix: pass steps.pr.conclusion into set_implement_outputs and factor it into the status determination.

[fantapop]

Bug: summary output is always empty (AUTOSOLVE_TMPDIR not shared between steps)

The assessment summary (and likely the implementation summary too) is always empty because AUTOSOLVE_TMPDIR is not shared across composite action steps.

Each step in a composite action runs in a new shell. run_step.sh creates AUTOSOLVE_TMPDIR and exports it, but that export only lives within that shell process:

# run_step.sh
if [ -z "${AUTOSOLVE_TMPDIR:-}" ]; then
  AUTOSOLVE_TMPDIR="$(mktemp -d "${TMPDIR:-/tmp}/autosolve_XXXXXX")"
  export AUTOSOLVE_TMPDIR
fi

So what happens:

1. run_assessment step creates /tmp/autosolve_abc123/, writes assessment_result.txt there
2. set_assess_outputs step starts a new shell, AUTOSOLVE_TMPDIR is empty, creates /tmp/autosolve_xyz789/, can't find assessment_result.txt → summary is empty

Observed output:

Assessment: PROCEED
Summary:

The assessment value itself works because it's passed via GITHUB_OUTPUT step outputs, not the temp file. But summary and result depend on reading from the temp dir.

Possible fix: write AUTOSOLVE_TMPDIR to GITHUB_ENV in run_step.sh so it persists across steps:

if [ -z "${AUTOSOLVE_TMPDIR:-}" ]; then
  AUTOSOLVE_TMPDIR="$(mktemp -d "${TMPDIR:-/tmp}/autosolve_XXXXXX")"
  export AUTOSOLVE_TMPDIR
  echo "AUTOSOLVE_TMPDIR=$AUTOSOLVE_TMPDIR" >> "${GITHUB_ENV:-/dev/null}"
fi

[fantapop]

Docs: callers should checkout the PR base branch, not the trigger ref

The autosolve actions work on whatever is already checked out. When a caller uses workflow_dispatch and runs from a non-default branch, actions/checkout checks out that branch by default. The autosolve action then branches from there, and the resulting PR includes unrelated commits from the triggering branch — not just Claude's changes.

For example, if the workflow runs from branch test-autosolve (which has a workflow file change), the PR against main includes both the workflow change and Claude's work.

This also causes a downstream problem: if the branch has workflow file modifications relative to the fork's main, GitHub requires the workflow scope on the push PAT — even though Claude never touched workflow files. Checking out the base branch avoids this entirely.

The fix in the caller workflow is simple:

- uses: actions/checkout@v5
  with:
    ref: main  # checkout the PR base branch, not the trigger ref
    fetch-depth: 0
    persist-credentials: false  # prevent checkout's credential helper from interfering with fork push

Worth documenting in the README examples, especially for workflow_dispatch use cases. The issues: [labeled] trigger doesn't have this problem since it always runs on the default branch.

Also noting that persist-credentials: false on the checkout step is important — without it, the checkout action's credential helper can interfere with the fork push token's credential helper set up by implement.sh.