Skip to content

feat(ci): add Bedrock integration tests with record/replay #4292

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
skamenan7 wants to merge 7 commits into
base: main
Choose a base branch
from
Open

feat(ci): add Bedrock integration tests with record/replay #4292

skamenan7 wants to merge 7 commits into from

Conversation

@skamenan7
Copy link
Contributor

@skamenan7 skamenan7 commented Dec 3, 2025

What does this PR do?

Adds Bedrock integration tests to CI using a record/replay mechanism. Tests run against pre-recorded API responses, without the need for AWS credentials in CI.

The main challenge was that Bedrock's OpenAI-compatible API doesn't support everything - no tool calling, no embeddings, no dynamic model listing. So instead of running the full base suite (which would fail on ~40 tests), I created a dedicated bedrock suite with just the tests that actually work.

Changes:

  • New run-bedrock.yaml stack config with dummy API key for replay mode
  • bedrock suite in suites.py pointing to 3 test functions (6 parametrized tests total)
  • Config resolution fix for distro::file.yaml format in library mode
  • Added a docs/source/providers/inference/bedrock_recording_guide.md so contributors with AWS access can re-record tests when needed

Closes #4095

Test Plan

Run from tests/integration/inference:


  uv run pytest -v \
    test_openai_completion.py::test_openai_chat_completion_non_streaming \
    test_openai_completion.py::test_openai_chat_completion_streaming \
    test_openai_completion.py::test_inference_store \
    --setup=bedrock \
    --stack-config=ci-tests::run-bedrock.yaml \
    --inference-mode=replay \
    -k "client_with_models"

Expected: 6 passed

test_openai_completion.py::test_openai_chat_completion_non_streaming[client_with_models-txt=bedrock/openai.gpt-oss-20b-1:0-inference:chat_com
 pletion:non_streaming_01] PASSED [ 16%]
 test_openai_completion.py::test_openai_chat_completion_streaming[client_with_models-txt=bedrock/openai.gpt-oss-20b-1:0-inference:chat_complet
 ion:streaming_01] PASSED [ 33%]
 test_openai_completion.py::test_inference_store[client_with_models-txt=bedrock/openai.gpt-oss-20b-1:0-True] PASSED                          [
  50%]
 test_openai_completion.py::test_openai_chat_completion_non_streaming[client_with_models-txt=bedrock/openai.gpt-oss-20b-1:0-inference:chat_com
 pletion:non_streaming_02] PASSED [ 66%]
 test_openai_completion.py::test_openai_chat_completion_streaming[client_with_models-txt=bedrock/openai.gpt-oss-20b-1:0-inference:chat_complet
 ion:streaming_02] PASSED [ 83%]
 test_openai_completion.py::test_inference_store[client_with_models-txt=bedrock/openai.gpt-oss-20b-1:0-False] PASSED
 [100%]

@meta-cla meta-cla bot added the CLA Signed This label is managed by the Meta Open Source bot. label Dec 3, 2025
@skamenan7 skamenan7 force-pushed the feat/bedrock-ci-4095 branch 7 times, most recently from 305be8e to 0667de8 Compare December 3, 2025 22:46
derekhiggins
derekhiggins reviewed Dec 4, 2025
View reviewed changes
@skamenan7 skamenan7 force-pushed the feat/bedrock-ci-4095 branch 3 times, most recently from 69eb2c4 to 51121ae Compare December 4, 2025 20:13
@skamenan7 skamenan7 marked this pull request as ready for review December 4, 2025 21:36
@skamenan7 skamenan7 force-pushed the feat/bedrock-ci-4095 branch 3 times, most recently from bd9f2dd to d75be15 Compare December 9, 2025 14:35
@mergify
Copy link

mergify bot commented Dec 10, 2025

This pull request has merge conflicts that must be resolved before it can be merged. @skamenan7 please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Dec 10, 2025
@skamenan7 skamenan7 force-pushed the feat/bedrock-ci-4095 branch 2 times, most recently from 2d82f01 to 5067cee Compare December 10, 2025 14:16
@mergify mergify bot removed the needs-rebase label Dec 10, 2025
@skamenan7 skamenan7 force-pushed the feat/bedrock-ci-4095 branch 6 times, most recently from 1d07a5a to 454aefe Compare December 16, 2025 15:32
derekhiggins
derekhiggins suggested changes Dec 16, 2025
View reviewed changes
@skamenan7 skamenan7 force-pushed the feat/bedrock-ci-4095 branch 4 times, most recently from 1db1c2b to 58fe994 Compare December 18, 2025 16:12
@skamenan7 skamenan7 force-pushed the feat/bedrock-ci-4095 branch 2 times, most recently from 1d92f48 to f3b96b6 Compare December 19, 2025 18:19
@skamenan7
Copy link
Contributor Author

skamenan7 commented Dec 19, 2025

The failing tests (ollama, gpt, backward-compat) are caused by a test infrastructure issue with stream_options hashing, not by Bedrock code. This is being fixed in #4326. All Bedrock-specific tests pass. Will rebase after #4326 merges.

@skamenan7 skamenan7 force-pushed the feat/bedrock-ci-4095 branch 2 times, most recently from 4fb5b72 to a0e27f0 Compare December 22, 2025 18:24
@skamenan7
Copy link
Contributor Author

skamenan7 commented Dec 22, 2025

Please review. CI is now green. cc:@derekhiggins and @cdoern

image

@skamenan7 skamenan7 force-pushed the feat/bedrock-ci-4095 branch from a0e27f0 to ea04e72 Compare January 5, 2026 13:55
skamenan7 and others added 7 commits January 5, 2026 08:57
@skamenan7
Add Bedrock CI integration with OpenAI-compatible endpoint
2513d6e 
- Create dedicated bedrock suite with 3 compatible test functions
- Add run-bedrock.yaml stack config for CI
- Enable config resolution for distro::file.yaml format in library mode
- Add test recordings for streaming, non-streaming, and inference store
- Skip tool-calling tests for Bedrock (not supported by AWS)
- Add recording guide documentation for contributors

Tested with GPT-OSS model on us-west-2 region.
@skamenan7
fix(ci): handle pytest node IDs in suite root filtering
23bd8cb 
The bedrock suite uses specific test function paths like
"test_file.py::test_function" in its roots. The pytest_ignore_collect
hook was treating these as filesystem paths, causing 0 tests to be
collected.

Changes:
- Strip "::test_function" suffix when checking file paths
- Add pytest_collection_modifyitems to filter to specific tests

Without this fix, cleanup_recordings.py marks bedrock recordings as
unused and deletes them in CI.
@skamenan7
refactor(ci): consolidate bedrock config into existing run.yaml
07d6c7f 
- Remove separate run-bedrock.yaml in favor of modifying templates
- Update bedrock provider config: dummy API key for replay mode, us-west-2 region
- Pre-register bedrock model in ci-tests template (Bedrock /v1/models returns empty)
- Update ci_matrix.json to use default stack config
@skamenan7
test: update bedrock config test to match new defaults
e170ba7
@skamenan7
docs(bedrock): fix config references and documentation issues
e78e6f4 
Consolidated config file usage (ci-tests::run.yaml instead of
run-bedrock.yaml), added Stack Configuration Choices section,
fixed default region to us-west-2, and updated pytest commands
to use full paths with clarification about test count.
@skamenan7
test: update bedrock config test to match new defaults
9130e01 
Changed default region from us-east-2 to us-west-2 to reflect where
GPT-OSS model is available. Update test expectation to match.
@skamenan7 @claude
fix(docs): correct config references and test count in bedrock docs
2b6aaa2 
- Replace non-existent run-bedrock.yaml with config.yaml in recording guide
- Replace non-existent run.yaml with config.yaml in record-replay.mdx
- Fix test count from "4 parametrized test cases" to "6 parametrized tests"

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@skamenan7 skamenan7 force-pushed the feat/bedrock-ci-4095 branch from ea04e72 to 2b6aaa2 Compare January 5, 2026 16:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ashwinb ashwinb Awaiting requested review from ashwinb ashwinb is a code owner

@raghotham raghotham Awaiting requested review from raghotham raghotham is a code owner

@ehhuang ehhuang Awaiting requested review from ehhuang ehhuang is a code owner

@leseb leseb Awaiting requested review from leseb leseb is a code owner

@bbrowning bbrowning Awaiting requested review from bbrowning bbrowning is a code owner

@mattf mattf Awaiting requested review from mattf mattf is a code owner

@franciscojavierarceo franciscojavierarceo Awaiting requested review from franciscojavierarceo franciscojavierarceo is a code owner

@cdoern cdoern Awaiting requested review from cdoern cdoern is a code owner

@derekhiggins derekhiggins Awaiting requested review from derekhiggins

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add Bedrock Provider to CI Integration Tests

2 participants

@skamenan7 @derekhiggins