Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions docs/agents.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,21 @@ Two distinct concepts share the name:

`agents try` therefore has no `--dry-run` flag — the whole command IS the dry-run. To preview the wire body without calling the backend, marshal `{"messages":[...], "configuration":{...}}` yourself. The dry-run e2e regression-asserts that `agents try --dry-run` is rejected.

## Providers: `-F` vs flags

`agents providers create` accepts either:

- **`-F <file>`** — full `ProviderAuthenticationCreate` JSON (all `providerName` variants, including `azure_openai` and `openai_compatible`), or
- **Flags** — `--name`, `--provider` (`openai` \| `anthropic` \| `google_genai` \| `deepseek`), plus exactly one of `--api-key`, `--api-key-stdin`, or `--api-key-env <VAR>`. Optional `--base-url` only for `openai` / `anthropic`.

`-F` and the shortcut flags are **mutually exclusive**.

`agents providers update <id>` accepts **`-F`** (patch JSON) **or** shortcut flags: any non-empty combination of `--name`, `--api-key` / `--api-key-stdin` / `--api-key-env`, and `--base-url`, with the same exclusivity rule against `-F`.

Prefer **`--api-key-env`** or **`--api-key-stdin`** over **`--api-key`** (shell history). `--dry-run` still shows the resolved body unredacted so authors can verify what would be sent.

Team sign-off and **Anya** QA checklist: [`docs/qa/arg_friendly_providers_SIGNOFF.md`](qa/arg_friendly_providers_SIGNOFF.md).

## Secret masking

`apiKey` (provider input) and `value` (secret-keys) are masked to `"***"` by default. Pass `--show-secret` to render verbatim. Masking happens at the cmd layer (`pkg/cmd/agents/shared/mask.go`), not the client. `--dry-run` does **not** mask: the user authored the file and is being shown what THEY are about to send. Three asterisks, no last-N preview — goal is "impossible to copy by accident", not "allow last-4 lookup". `secretFieldNames` is the closed set; extend alphabetically when new credential fields land.
Expand Down
146 changes: 146 additions & 0 deletions docs/qa/arg_friendly_providers_SIGNOFF.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,146 @@
# Arg-friendly `agents providers` — team sign-off & QA sheet

**Feature branch:** `feat/agents-providers-arg-friendly` (stacked on `lab_week_3` / [PR #212](https://github.com/algolia/cli/pull/212))

**Scope:** Optional flags for `agents providers create|update` so the common “single API key” case does not require `-F`. Full JSON via `-F` unchanged; Azure / `openai_compatible` still require `-F`.

**Planning reference:** Lab Week 3 roles (“The team (5 people)”: Iris, Marek, Yuki, Diego, Priya) — often kept locally as `tmp/lab_week_3_plan.md` beside other lab notes.

> This file lives under `docs/qa/` so it ships with the repo. Some checkouts list `tmp/` in `.git/info/exclude`; use this path as the canonical copy for PRs.

---

## Engineering sign-off (planning team)

| Person | Role | Position on this change |
|--------|------|-------------------------|
| **Iris** | Auth & identity | **On board.** Out of scope for OAuth; provider keys stay user-supplied. Flags do not change header auth (`X-Algolia-Application-Id` / `X-Algolia-API-Key`). |
| **Marek** | CLI / DX | **On board.** Matches Cobra patterns: `-F` vs flags mutually exclusive; long-help documents shell-history risk for `--api-key` and recommends `--api-key-env` / `--api-key-stdin`. |
| **Yuki** | Agent Studio backend | **On board.** Wire body is identical to JSON create/patch; CLI only assembles JSON for `openai`, `anthropic`, `google_genai`, `deepseek`. |
| **Diego** | Security | **On board.** Same threat model as `-F`: secrets can still hit shell history if the user passes `--api-key`; env/stdin paths preferred. No new persistence. |
| **Priya** | Search UI (aspirational track) | **N/A / on board.** No UI work; no objection to CLI ergonomics that reduce temp files. |

---

## QA owner: **Anya**

**Exit criteria**

- [ ] `task build` (or `go build` per repo) produces a binary; all of the commands below use that binary as `./algolia`.
- [ ] Agents e2e suite runs with app credentials: `go test ./e2e -tags=e2e -run TestAgents -count=1` includes `testscripts/agents/providers.txtar` and passes (or document skip if creds unavailable).
- [ ] Unit tests: `go test ./pkg/cmd/agents/providers/...` green.
- [ ] Live smoke (staging / beta app): at least one **create** and one **update** via flags succeed; **dry-run** paths show expected JSON and do not call mutating APIs.

---

## Anya — testing command sheet

Run from the **repository root** after `task build`. Replace placeholders.

### 0. Binary and profile

```bash
cd /path/to/cli
task build
./algolia version
# Ensure profile or env has app + key (same as other `algolia agents` commands)
```

### 1. Unit tests (no network)

```bash
go test ./pkg/cmd/agents/providers/... -count=1
```

### 2. Contract tests (e2e harness; needs credentials)

```bash
export ALGOLIA_APPLICATION_ID="YOUR_APP_ID"
export ALGOLIA_API_KEY="YOUR_API_KEY"
go test ./e2e -tags=e2e -run TestAgents -count=1
```

### 3. Help / usage sanity

```bash
./algolia agents providers create --help
./algolia agents providers update --help
```

### 4. Create — **dry-run** (flags), no API call

```bash
./algolia agents providers create \
--name "qa-cli-flags-smoke" \
--provider anthropic \
--api-key "sk-ant-REDACTED" \
--dry-run
```

Expect: human or structured dry-run summary; body includes `providerName`, `name`, `input.apiKey` (unmasked in dry-run per masking rules in `docs/agents.md`).

### 5. Create — **dry-run** via env (no key on command line)

```bash
export QA_ANTHROPIC_KEY="sk-ant-your-real-test-key"
./algolia agents providers create \
--name "qa-cli-flags-env" \
--provider anthropic \
--api-key-env QA_ANTHROPIC_KEY \
--dry-run
unset QA_ANTHROPIC_KEY
```

### 6. Create — **live** (optional; consumes provider quota)

```bash
export QA_ANTHROPIC_KEY="sk-ant-..."
PID=$(./algolia agents providers create \
--name "qa-live-$(date +%s)" \
--provider anthropic \
--api-key-env QA_ANTHROPIC_KEY \
--output json | jq -r .id)
echo "PID=$PID"
```

### 7. Update — **dry-run** (rename only)

```bash
./algolia agents providers update "$PID" --name "qa-renamed" --dry-run
```

### 8. Update — **live** (rotate key or rename)

```bash
./algolia agents providers update "$PID" --name "qa-renamed-final"
# or rotate:
# ./algolia agents providers update "$PID" --api-key-env QA_ANTHROPIC_KEY
```

### 9. Negative cases

```bash
# Must error: cannot mix -F and flags (use a real JSON file path):
# echo '{"name":"x","providerName":"openai","input":{"apiKey":"sk"}}' > /tmp/prov.json
# ./algolia agents providers create -F /tmp/prov.json --name x --provider openai --api-key sk 2>&1 | head -5

# Must error: unsupported provider for flag path
./algolia agents providers create --name x --provider azure_openai --api-key sk --dry-run 2>&1 | head -5

# Must error: create without -F and without full flag set
./algolia agents providers create 2>&1 | head -5
```

### 10. Cleanup

```bash
./algolia agents providers delete "$PID" -y
```

---

## Report back (local only — do **not** commit)

Anya drops a short vet note under **`tmp/qa/anya_arg_friendly_providers_vet/`** (e.g. `REPORT.md`) for Catalin to read: date, commit SHA, pass/fail per section, backend 422 snippets if any.

That path stays **out of git** (scratch / personal checkout). Nothing from Anya’s QA report belongs in PR commits—only the feature and the in-repo docs above.
4 changes: 2 additions & 2 deletions e2e/testscripts/agents/providers.txtar
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,9 @@ stderr 'requires exactly 1 argument'
! exec algolia agents providers delete some-id
stderr '--confirm required'

# `create` requires --file
# `create` without -F or shortcut flags prints usage error
! exec algolia agents providers create
stderr 'required flag\(s\) "file" not set'
stderr 'specify a JSON body with -F, or pass --name, --provider, and an API key flag'

# --dry-run on create previews POST to /1/providers, no network
exec algolia agents providers create -F spec.json --dry-run
Expand Down
73 changes: 61 additions & 12 deletions pkg/cmd/agents/providers/create.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,12 @@ type CreateOptions struct {
PrintFlags *cmdutil.PrintFlags

File string
Name string
Provider string
APIKey string
APIKeyStdin bool
APIKeyEnv string
BaseURL string
DryRun bool
Show bool
OutputChanged bool
Expand All @@ -35,20 +41,30 @@ func newCreateCmd(f *cmdutil.Factory, runF func(*CreateOptions) error) *cobra.Co
}

cmd := &cobra.Command{
Use: "create -F <file>",
Short: "Create an LLM provider authentication from a JSON file",
Use: "create (-F <file> | --name <name> --provider <type> " +
"(--api-key <key> | --api-key-stdin | --api-key-env <var>))",
Short: "Create an LLM provider authentication",
Long: heredoc.Doc(`
Create a provider authentication from a JSON file describing
the ProviderAuthenticationCreate body (name, providerName,
input). The "input" subobject's shape varies per providerName:
Create a provider authentication from a JSON file (-F) or from
flags for the common case (OpenAI, Anthropic, Google GenAI, or
DeepSeek with a single API key).

The -F body is the ProviderAuthenticationCreate JSON (name,
providerName, input). The "input" subobject's shape varies per
providerName:

- openai / anthropic: {apiKey, baseUrl?}
- azure_openai: {apiKey, azureEndpoint, azureDeployment, apiVersion?}
- openai_compatible: {apiKey, baseUrl, defaultModel}
- google_genai / deepseek: {apiKey}

The file is sent verbatim; field-level validation is the
backend's job (a 4xx surfaces with the structured detail).
With flags, only openai, anthropic, google_genai, and deepseek
are supported; use -F for Azure or openai_compatible.

Do not combine -F with --name/--provider/--api-key*.

Prefer --api-key-stdin or --api-key-env over --api-key (shell
history may record raw flags).

Use --dry-run to preview the request without sending.

Expand All @@ -57,6 +73,7 @@ func newCreateCmd(f *cmdutil.Factory, runF func(*CreateOptions) error) *cobra.Co
`),
Example: heredoc.Doc(`
$ algolia agents providers create -F openai-prod.json
$ algolia agents providers create --name openai-prod --provider openai --api-key-env OPENAI_API_KEY
$ cat spec.json | algolia agents providers create -F -
$ algolia agents providers create -F spec.json --dry-run
`),
Expand All @@ -73,7 +90,12 @@ func newCreateCmd(f *cmdutil.Factory, runF func(*CreateOptions) error) *cobra.Co

cmd.Flags().
StringVarP(&opts.File, "file", "F", "", "JSON file with the provider body (use \"-\" for stdin)")
_ = cmd.MarkFlagRequired("file")
cmd.Flags().StringVar(&opts.Name, "name", "", "Provider label (shortcut; not with -F)")
cmd.Flags().StringVar(&opts.Provider, "provider", "", `Provider backend: openai, anthropic, google_genai, or deepseek (shortcut; not with -F)`)
cmd.Flags().StringVar(&opts.APIKey, "api-key", "", "API credential (shortcut; not with -F)")
cmd.Flags().BoolVar(&opts.APIKeyStdin, "api-key-stdin", false, "Read API key from stdin (shortcut; not with -F)")
cmd.Flags().StringVar(&opts.APIKeyEnv, "api-key-env", "", "Read API key from this environment variable (shortcut; not with -F)")
cmd.Flags().StringVar(&opts.BaseURL, "base-url", "", `Optional OpenAI / Anthropic-compatible base URL (shortcut; not with -F)`)
cmd.Flags().
BoolVar(&opts.DryRun, "dry-run", false, "Validate and print the resolved request body without calling the API")
cmd.Flags().
Expand All @@ -83,14 +105,41 @@ func newCreateCmd(f *cmdutil.Factory, runF func(*CreateOptions) error) *cobra.Co
}

func runCreateCmd(opts *CreateOptions) error {
body, err := shared.ReadJSONFile(opts.IO.In, opts.File)
if err != nil {
return err
source := opts.File

var body json.RawMessage
var err error

switch {
case inlineFlagsConflictWithFile(opts.File, opts.Name, opts.Provider, opts.APIKey, opts.APIKeyEnv, opts.BaseURL, opts.APIKeyStdin):
return cmdutil.FlagErrorf("cannot combine -F/--file with --name/--provider/--api-key/--api-key-* flags")
case opts.File != "":
body, err = shared.ReadJSONFile(opts.IO.In, opts.File)
if err != nil {
return err
}
default:
if !createShortcutAttempted(opts.Name, opts.Provider, opts.APIKey, opts.APIKeyEnv, opts.BaseURL, opts.APIKeyStdin) {
return cmdutil.FlagErrorf(`specify a JSON body with -F, or pass --name, --provider, and an API key flag`)
}
if opts.Name == "" || opts.Provider == "" {
return cmdutil.FlagErrorf("when creating without -F, both --name and --provider are required")
}
key, err := resolveProvidedAPIKey(opts.IO.In, opts.APIKey, opts.APIKeyEnv, opts.APIKeyStdin)
if err != nil {
return err
}
raw, err := marshalSimpleProviderCreate(opts.Name, opts.Provider, opts.BaseURL, key)
if err != nil {
return err
}
body = raw
source = "(flags)"
}

if opts.DryRun {
return shared.PrintDryRun(opts.IO, opts.PrintFlags, opts.OutputChanged,
"create_provider", "POST /1/providers", opts.File, body, nil)
"create_provider", "POST /1/providers", source, body, nil)
}

client, err := opts.AgentStudioClient()
Expand Down
59 changes: 59 additions & 0 deletions pkg/cmd/agents/providers/create_test.go
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
package providers

import (
"bytes"
"io"
"net/http"
"net/http/httptest"
"testing"
Expand Down Expand Up @@ -74,3 +76,60 @@ func Test_runCreateCmd_RejectsInvalidJSON(t *testing.T) {
require.Error(t, err)
assert.Contains(t, err.Error(), "not valid JSON")
}

func Test_runCreateCmd_Flags_PostsMinimalOpenAIBody(t *testing.T) {
mux := http.NewServeMux()
mux.HandleFunc("/1/providers", func(w http.ResponseWriter, r *http.Request) {
assert.Equal(t, http.MethodPost, r.Method)
body, err := io.ReadAll(r.Body)
require.NoError(t, err)
assert.JSONEq(t, `{"input":{"apiKey":"sk-env"},"name":"prod","providerName":"openai"}`,
string(bytes.TrimSpace(body)))
w.WriteHeader(http.StatusCreated)
_, _ = w.Write([]byte(`{
"id":"p1","name":"prod","providerName":"openai",
"input":{"apiKey":"sk-env"},
"createdAt":"2026-01-01T00:00:00Z","updatedAt":"2026-01-01T00:00:00Z"
}`))
})
ts := httptest.NewServer(mux)
t.Cleanup(ts.Close)

t.Setenv("OPENAI_CLI_TEST_KEY", "sk-env")

f, out := test.NewFactory(false, nil, nil, "")
f.AgentStudioClient = sharedtest.NewClient(t, ts)

cmd := NewProvidersCmd(f)
cli := `create --name prod --provider openai --api-key-env OPENAI_CLI_TEST_KEY`
result, err := test.Execute(cmd, cli, out)
require.NoError(t, err)
assert.Contains(t, result.String(), `"name":"prod"`)
}

func Test_runCreateCmd_FlagsMutuallyExclusiveWithFile(t *testing.T) {
specPath := sharedtest.WriteTempJSON(
t,
"spec.json",
`{"name":"x","providerName":"openai","input":{"apiKey":"sk"}}`,
)
f, out := test.NewFactory(false, nil, nil, "")
cmd := NewProvidersCmd(f)
cli := "create --name clash --provider openai --api-key sk -F " + specPath
_, err := test.Execute(cmd, cli, out)
require.Error(t, err)
assert.Contains(t, err.Error(), "combine")
}

func Test_runCreateCmd_FlagsUnsupportedProviderUsesF(t *testing.T) {
f, out := test.NewFactory(false, nil, nil, "")
cmd := NewProvidersCmd(f)
_, err := test.Execute(
cmd,
`create --name azure --provider azure_openai --api-key sk-azure --dry-run`,
out,
)
require.Error(t, err)
assert.Contains(t, err.Error(), "use -F")
}

Loading