Skip to content

Commit ab41232

Browse files
lforstlforst
authored
chore: Add nightly claude CI job to identify SDK support gaps (#1619)
Just a small experiment to automate staying on top of things.
1 parent 5557eb2 commit ab41232

1 file changed

Lines changed: 120 additions & 0 deletions

File tree

‎.github/workflows/agent-automations.yaml‎

Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
name: Agent Automation
2+
3+
on:
4+
schedule:
5+
- cron: "0 6 * * *"
6+
workflow_dispatch:
7+
8+
permissions:
9+
contents: read
10+
11+
concurrency:
12+
group: provider-gap-audit-${{ github.ref }}
13+
cancel-in-progress: false
14+
15+
jobs:
16+
audit-sdk-coverage:
17+
name: Audit SDK Coverage
18+
runs-on: ubuntu-latest
19+
timeout-minutes: 45
20+
steps:
21+
- name: Generate GitHub App token
22+
id: app-token
23+
uses: actions/create-github-app-token@f8d387b68d61c58ab83c6c016672934102569859 # v3.0.0
24+
with:
25+
app-id: ${{ secrets.BRAINTRUST_BOT_APP_ID }}
26+
private-key: ${{ secrets.BRAINTRUST_BOT_PRIVATE_KEY }}
27+
owner: braintrustdata
28+
repositories: |
29+
braintrust-sdk-javascript
30+
permission-contents: read
31+
permission-issues: write
32+
33+
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
34+
with:
35+
persist-credentials: false
36+
37+
- name: Run Claude provider gap audit
38+
uses: anthropics/claude-code-action@df37d2f0760a4b5683a6e617c9325bc1a36443f6 # v1.0.75
39+
with:
40+
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
41+
# Pass our own app token so the Claude action does not try to mint its
42+
# own GitHub token via OIDC. That OIDC path currently fails for this
43+
# workflow shape upstream, and this app token also lets us tightly scope
44+
# Claude's remote permissions to repo contents read + issues write only.
45+
github_token: ${{ steps.app-token.outputs.token }}
46+
prompt: |
47+
# Goal
48+
49+
Find important instrumentation gaps in this repository's AI-facing integrations.
50+
51+
A gap means the upstream SDK or framework clearly supports something that this repository does not yet instrument, or instruments with materially less detail.
52+
53+
# Scope
54+
55+
- Infer the relevant surfaces from the checked-out repository itself.
56+
- Focus on repo-owned AI instrumentation surfaces with meaningful upstream API, docs, and release surfaces.
57+
- Ignore generic runtime or infrastructure integrations that do not have an upstream AI API surface to compare against.
58+
- Do not open parity issues for deprecated or no-op surfaces unless you find real deprecation drift or docs drift worth reporting.
59+
60+
# Process
61+
62+
1. Inspect local code, tests, docs, examples, and e2e scenarios to understand what is already instrumented.
63+
2. For each relevant surface, independently discover the current official upstream docs and recent official releases or changelogs.
64+
3. Compare current upstream capabilities to current Braintrust instrumentation in this repo.
65+
4. Always check the latest Braintrust docs at https://www.braintrust.dev/docs before deciding how to describe a gap.
66+
5. Search existing GitHub issues for duplicates before creating anything.
67+
6. Only act on high-confidence, concrete gaps tied to missing APIs, unsupported call patterns, or missing instrumentation detail.
68+
69+
# Examples
70+
71+
## Good
72+
73+
- The upstream SDK now has a stable `responses.stream()` helper, but this repo does not instrument it at all, or instruments it without final result metadata that other similar APIs already capture here.
74+
- This repo instruments a provider's basic text generation API, but not its newer tool-calling or agent tracing API even though that API is now official and documented.
75+
76+
## Bad
77+
78+
- A vague suspicion that "something in streaming may be missing" without a concrete upstream API and a concrete repo gap.
79+
- Opening a separate issue for every release note bullet when they all describe the same missing instrumentation area.
80+
81+
# If You Find Actionable Non-Duplicate Gaps
82+
83+
- Create at most 5 issues in this run.
84+
- Create one issue per distinct gap.
85+
- Keep each issue concise, concrete, and source-backed.
86+
- Include a hidden marker comment near the top of the issue body in this exact form:
87+
88+
```html
89+
<!-- provider-gap-audit: <gap_id> -->
90+
```
91+
92+
Each issue should clearly include:
93+
94+
- what instrumentation is missing
95+
- whether Braintrust docs suggest the capability is `supported`, `unclear`, or `not_found`
96+
- exact upstream sources
97+
- exact Braintrust docs source or sources
98+
- exact local repo files you inspected
99+
100+
# Duplicate Handling
101+
102+
- Do not create an issue if an open issue already covers the same gap.
103+
- Treat a matching hidden marker comment or a clearly equivalent open issue as a duplicate.
104+
- If duplicate checking is inconclusive, do not create the issue.
105+
106+
# Constraints
107+
108+
- Discover source URLs yourself. Do not rely on a preset list.
109+
- Prefer official docs and official release sources.
110+
- Do not create comments.
111+
- Do not update, close, or label existing issues.
112+
- Do not create pull requests.
113+
- If there are no high-confidence non-duplicate gaps, do nothing.
114+
# The Claude action includes base GitHub tools by default, and --allowedTools adds to that set
115+
# rather than replacing it. Keep the deny-list so Claude cannot use other remote write tools.
116+
claude_args: |
117+
--model claude-opus-4-6
118+
--max-turns 20
119+
--allowedTools "Read,Glob,Grep,LS,WebSearch,WebFetch,mcp__github__get_issue,mcp__github__get_issue_comments,mcp__github__search_issues,mcp__github__list_issues,mcp__github__create_issue"
120+
--disallowedTools "Bash,Edit,MultiEdit,Write,Replace,NotebookEditCell,mcp__github__create_issue_comment,mcp__github__update_issue,mcp__github__create_pr,mcp__github__create_or_update_file,mcp__github__delete_file,mcp__github_file_ops__commit_files,mcp__github_file_ops__delete_files"

0 commit comments

Comments
 (0)