feat: add azd CLI evaluation and testing framework#7202

Merged

spboyer merged 28 commits intomainfrom

feat/eval-framework

Apr 14, 2026

Commits on Apr 13, 2026

feat: add azd CLI evaluation and testing framework

spboyer
and
Copilot
committed
docs: add authentication and secrets section to eval README

spboyer
and
Copilot
committed
docs: add comprehensive how-to guides for creating evals, graders, and tests

spboyer
and
Copilot
committed
fix: resolve CI failures in eval unit tests and cspell

spboyer
and
Copilot
committed
fix: stop command-sequencing tests from overriding AZD_CONFIG_DIR

spboyer
and
Copilot
committed
docs: expand auth section with subscription config and no-popup guarantee

spboyer
and
Copilot
committed
refactor: address review feedback from @jongio and Copilot

spboyer
and
Copilot
committed
fix: address round 2 review feedback from @jongio

spboyer
and
Copilot
committed
Fix review: mock targets, env remove, CI grader tests, HTTPError handling

spboyer
and
Copilot
committed
fix: skip waza validation when CLI not installed

spboyer
and
Copilot
committed
Update cli/azd/test/eval/tasks/environment/switch-env.yaml

spboyer
and
jongio
committed
Update cli/azd/test/eval/README.md

spboyer
and
jongio
committed
fix: address remaining PR review feedback

spboyer
and
Copilot
committed
chore: add 40-minute timeout to eval-e2e lifecycle step

spboyer
and
Copilot
committed
Address review feedback on eval framework

spboyer
and
Copilot
committed
Remove continue-on-error masking in eval-waza.yml

spboyer
and
Copilot
committed
Fix error handling in graders and cleanup step

spboyer
and
Copilot
committed
Address review feedback on eval framework

spboyer
and
Copilot
committed
Address review feedback on eval framework (round 2)

spboyer
and
Copilot
committed
Re-trigger CI (flaky TestInitializer_PromptIfNonEmpty)

spboyer
and
Copilot
committed
Use tag-for-deletion pattern in eval cleanup step

spboyer
and
Copilot
committed
Address review feedback: timeout, imports, pipeline command

spboyer
and
Copilot
committed
fix: remove always-skipped waza step from CI workflow

spboyer
and
Copilot
committed
chore: apply go fix modernization (strings.SplitSeq)

spboyer
and
Copilot
committed
Migrate eval CI from GitHub Actions to ADO pipelines

spboyer
and
Copilot
committed
fix: remove continueOnError from eval step in eval-e2e pipeline

spboyer
and
Copilot
committed
fix: address review feedback on eval framework PR

spboyer
and
Copilot
committed

Commits on Apr 14, 2026

fix: scope resource group cleanup to current build ID

spboyer
and
Copilot
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add azd CLI evaluation and testing framework#7202

feat: add azd CLI evaluation and testing framework#7202
spboyer merged 28 commits intomainfrom
feat/eval-framework

Commits on Apr 13, 2026

feat: add azd CLI evaluation and testing framework

docs: add authentication and secrets section to eval README

docs: add comprehensive how-to guides for creating evals, graders, and tests

fix: resolve CI failures in eval unit tests and cspell

fix: stop command-sequencing tests from overriding AZD_CONFIG_DIR

docs: expand auth section with subscription config and no-popup guarantee

refactor: address review feedback from @jongio and Copilot

fix: address round 2 review feedback from @jongio

Fix review: mock targets, env remove, CI grader tests, HTTPError handling

fix: skip waza validation when CLI not installed

Update cli/azd/test/eval/tasks/environment/switch-env.yaml

Update cli/azd/test/eval/README.md

fix: address remaining PR review feedback

chore: add 40-minute timeout to eval-e2e lifecycle step

Address review feedback on eval framework

Remove continue-on-error masking in eval-waza.yml

Fix error handling in graders and cleanup step

Address review feedback on eval framework

Address review feedback on eval framework (round 2)

Re-trigger CI (flaky TestInitializer_PromptIfNonEmpty)

Use tag-for-deletion pattern in eval cleanup step

Address review feedback: timeout, imports, pipeline command

fix: remove always-skipped waza step from CI workflow

chore: apply go fix modernization (strings.SplitSeq)

Migrate eval CI from GitHub Actions to ADO pipelines

fix: remove continueOnError from eval step in eval-e2e pipeline

fix: address review feedback on eval framework PR

Commits on Apr 14, 2026

fix: scope resource group cleanup to current build ID