Skip to content

Commit 265a462

Browse files
AbhiPrasadAbhiPrasad
authored
chore: Add nightly claude CI job to identify SDK support gaps (#104)
1 parent 42bed48 commit 265a462

1 file changed

Lines changed: 128 additions & 0 deletions

File tree

‎.github/workflows/nightly-sdk-gap-audit.yaml‎

Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
name: Nightly SDK Gap Audit
2+
3+
on:
4+
schedule:
5+
- cron: "0 6 * * *"
6+
workflow_dispatch:
7+
8+
permissions:
9+
contents: read
10+
11+
concurrency:
12+
group: provider-gap-audit-${{ github.ref }}
13+
cancel-in-progress: false
14+
15+
jobs:
16+
audit-sdk-coverage:
17+
name: Audit SDK Coverage
18+
runs-on: ubuntu-latest
19+
timeout-minutes: 45
20+
steps:
21+
- name: Generate GitHub App token
22+
id: app-token
23+
uses: actions/create-github-app-token@f8d387b68d61c58ab83c6c016672934102569859 # v3.0.0
24+
with:
25+
app-id: ${{ secrets.BRAINTRUST_BOT_APP_ID }}
26+
private-key: ${{ secrets.BRAINTRUST_BOT_PRIVATE_KEY }}
27+
owner: braintrustdata
28+
repositories: |
29+
braintrust-sdk-python
30+
permission-contents: read
31+
permission-issues: write
32+
33+
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
34+
with:
35+
persist-credentials: false
36+
37+
- name: Run Claude provider gap audit
38+
uses: anthropics/claude-code-action@df37d2f0760a4b5683a6e617c9325bc1a36443f6 # v1.0.75
39+
with:
40+
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
41+
# Pass our own app token so the Claude action does not try to mint its
42+
# own GitHub token via OIDC. That OIDC path currently fails for this
43+
# workflow shape upstream, and this app token also lets us tightly scope
44+
# Claude's remote permissions to repo contents read + issues write only.
45+
github_token: ${{ steps.app-token.outputs.token }}
46+
prompt: |
47+
# Goal
48+
49+
Find important instrumentation gaps in this repository's AI-facing integrations.
50+
51+
A gap means the upstream SDK or framework clearly supports something that this repository does not yet instrument, or instruments with materially less detail.
52+
53+
# Scope
54+
55+
This repository is the Braintrust Python SDK. Focus exclusively on the Python integrations it ships.
56+
Discover them by inspecting the following directories in the checked-out repository:
57+
58+
- `py/src/braintrust/wrappers/` — AI provider wrappers and framework integrations
59+
- `py/src/braintrust/contrib/` — additional contributed integrations
60+
- `py/examples/` — usage examples that reveal what surfaces are considered supported
61+
62+
Do not look at the `integrations/` directory (deprecated old integrations).
63+
64+
- Infer additional relevant surfaces from the checked-out repository itself (tests, noxfile.py, pyproject.toml).
65+
- Ignore generic runtime or infrastructure integrations that do not have an upstream AI API surface to compare against.
66+
- Do not open parity issues for deprecated or no-op surfaces unless you find real deprecation drift or docs drift worth reporting.
67+
68+
# Process
69+
70+
1. Inspect local code, tests, docs, examples, and e2e scenarios to understand what is already instrumented.
71+
2. For each relevant surface, independently discover the current official upstream docs and recent official releases or changelogs.
72+
3. Compare current upstream capabilities to current Braintrust instrumentation in this repo.
73+
4. Always check the latest Braintrust docs at https://www.braintrust.dev/docs before deciding how to describe a gap.
74+
5. Search existing GitHub issues for duplicates before creating anything.
75+
6. Only act on high-confidence, concrete gaps tied to missing APIs, unsupported call patterns, or missing instrumentation detail.
76+
77+
# Examples
78+
79+
## Good
80+
81+
- The upstream SDK now has a stable `responses.stream()` helper, but this repo does not instrument it at all, or instruments it without final result metadata that other similar APIs already capture here.
82+
- This repo instruments a provider's basic text generation API, but not its newer tool-calling or agent tracing API even though that API is now official and documented.
83+
84+
## Bad
85+
86+
- A vague suspicion that "something in streaming may be missing" without a concrete upstream API and a concrete repo gap.
87+
- Opening a separate issue for every release note bullet when they all describe the same missing instrumentation area.
88+
89+
# If You Find Actionable Non-Duplicate Gaps
90+
91+
- Create at most 5 issues in this run.
92+
- Create one issue per distinct gap.
93+
- Keep each issue concise, concrete, and source-backed.
94+
- Include a hidden marker comment near the top of the issue body in this exact form:
95+
96+
```html
97+
<!-- provider-gap-audit: <gap_id> -->
98+
```
99+
100+
Each issue should clearly include:
101+
102+
- what instrumentation is missing
103+
- whether Braintrust docs suggest the capability is `supported`, `unclear`, or `not_found`
104+
- exact upstream sources
105+
- exact Braintrust docs source or sources
106+
- exact local repo files you inspected
107+
108+
# Duplicate Handling
109+
110+
- Do not create an issue if an open issue already covers the same gap.
111+
- Treat a matching hidden marker comment or a clearly equivalent open issue as a duplicate.
112+
- If duplicate checking is inconclusive, do not create the issue.
113+
114+
# Constraints
115+
116+
- Discover source URLs yourself. Do not rely on a preset list.
117+
- Prefer official docs and official release sources.
118+
- Do not create comments.
119+
- Do not update, close, or label existing issues.
120+
- Do not create pull requests.
121+
- If there are no high-confidence non-duplicate gaps, do nothing.
122+
# The Claude action includes base GitHub tools by default, and --allowedTools adds to that set
123+
# rather than replacing it. Keep the deny-list so Claude cannot use other remote write tools.
124+
claude_args: |
125+
--model claude-opus-4-6
126+
--max-turns 20
127+
--allowedTools "Read,Glob,Grep,LS,WebSearch,WebFetch,mcp__github__get_issue,mcp__github__get_issue_comments,mcp__github__search_issues,mcp__github__list_issues,mcp__github__create_issue"
128+
--disallowedTools "Bash,Edit,MultiEdit,Write,Replace,NotebookEditCell,mcp__github__create_issue_comment,mcp__github__update_issue,mcp__github__create_pr,mcp__github__create_or_update_file,mcp__github__delete_file,mcp__github_file_ops__commit_files,mcp__github_file_ops__delete_files"

0 commit comments

Comments
 (0)