Skip to content

Add generator script for Amazon Bedrock models#1431

Draft
athewsey wants to merge 1 commit intoanomalyco:devfrom
athewsey:feat/generate-bedrock
Draft

Add generator script for Amazon Bedrock models#1431
athewsey wants to merge 1 commit intoanomalyco:devfrom
athewsey:feat/generate-bedrock

Conversation

@athewsey
Copy link
Copy Markdown

Hi folks! For up-front transparency, I work at AWS - though not in the actual Bedrock product team.

I came across models.dev recently and noticed a couple of gaps & inconsistencies in the Bedrock model listings... so rather than fixing them as a one-off, this PR attempts to introduce an automatic generator script for Bedrock similar to the ones already set up for other providers.

Keen to get your initial feedback before I do the last manual tidy-ups, as mentioned below 🙏

Implementation overview

This script:

  1. Crawls the Bedrock ListFoundationModels API in a couple of regions to fetch the full list of supported models (some are supported in us-west-2 but not us-east-1, and vice-versa - but AFAIK those two regions should generally provide the complete list between them)
  2. Fetches pricing information from the AWS Pricing GetProducts API for the regions tested,
  3. Tries to match the pricing records to the listed models, which is unfortunately non-trivial due to the setup of the pricing API...
  4. Checks and merges changes to the TOMLs, similar to the existing generate-* scripts.

Since it calls APIs, users will need an AWS Account/credentials set up to run the generator - but there should be no cost associated with doing so, as it's just fetching listings.

Known to-dos

There's some outstanding manual work to go through the new model files, especially, and add in some missing information that's not available through the APIs this script uses (e.g. audio/pdf modality support, reasoning & tool-calling support, etc) - and cases where the price mapping has come up empty. I'm OK to do that, but wanted to show & get feedback on the automated portion first. I also figured delaying those edits would help show the limitations of what the current script is(n't) able to populate.

Key call-outs & questions

(See also the docstring at the top of generate-bedrock.ts)

  1. AWS API auth (sigv4) is nice and secure but unpleasantly complex to re-implement from scratch - so this PR introduces dependencies on the official @aws-sdk/client-bedrock and @aws-sdk/client-pricing libraries.
  2. As noted in Bedrock Per-Region Pricing #1006, Bedrock prices vary by AWS Region and inference tier, but the schema currently has no way to reflect multiple prices for a model.
    • This PR aims to standardize on the lowest available on-demand, online inference price - ignoring any volume discounts, commitments, or batch inference options - across the checked regions and service tiers... Which I think seems fair?
  3. ListFoundationModels returns separate results for models that support explicitly different IDs for different context lengths - which in some cases drives different pricing (e.g. "Claude Sonnet 4.5 - Long Context") and/or latency characteristics. This script has ended up creating quite a few new TOML files with context length suffixes e.g. :8k for these endpoints... Not sure if there's appetite for this, or we need to consolidate?
  4. Some formatting points like underscore thousands separators (e.g. 1_000_000), key ordering, etc. seemed to be inconsistently applied in the existing TOMLs... Of course it should be pretty easy to change how the script standardizes these if wanted.
  5. A lot of the extra information (like reasoning, temperature, etc) we could automatically validate with test model invocations - but I avoided this for now because then running the script would cost money.

Add (and run) a generator script for updating models on Amazon
Bedrock.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant