feat: OpenAI responses extractors

[eternalcuriouslearner]

Description

This change adds strongly typed Responses API extractors for openai-v2 and reduces dependence on Any, based on feedback we received on the sync Anthropic SDK instrumentation. Pydantic models are used
here to:

• normalize dict and object inputs
• avoid Any
• validate only the fields we care about
• fail safely by logging and returning empty values instead of breaking telemetry

Dependencies required for this change:

• pydantic >= 2, < 3

Fixes #3436 partly

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How Has This Been Tested?

Focused extractor tests were added/updated to cover:

• request validation failures returning empty values and logging
• response validation failures returning empty values and logging
• response usage extraction from object and mapping inputs
• capture_content=True populating output_messages
• prevalidated response models skipping revalidation
• Responses API status-to-finish-reason mapping behavior

Reproduce with:

python3 -m pytest --noconftest instrumentation-genai/opentelemetry-instrumentation-openai-v2/tests/test_response_extractors.py

Test result in local environment:

• Focused test_response_extractors.py passes (14 tests)

Does This PR Require a Core Repo Change?

• Yes. - Link to PR:
• No.

Checklist:

• Followed the style guidelines of this project
• Changelogs have been updated
• Unit tests have been added
• Documentation has been updated

[xrmx]

Question: isn't openai using pydantic itself and so aren't its answer expected to be of the expected shape?

[eternalcuriouslearner]

Question: isn't openai using pydantic itself and so aren't its answer expected to be of the expected shape?

Howdy Riccardo, yep OpenAI uses pydantic. We’re using Pydantic here to validate and normalize the minimal request/response shapes that the instrumentation depends on. The local models let us avoid Any, handle the supported inputs through one path, and fail closed by logging and returning empty values instead of letting extraction errors break telemetry. In a nutshell I am using pydantic for following reasons:

• normalize dict and object inputs
• avoid Any
• validate only the fields we care about
• fail safely by logging and returning empty values instead of breaking telemetry

Lmk your thoughts.

CC: @xrmx @aabmass

[lmolkova]

it seems openai already depends on pydantic, does it need to be an explicit required dependency here?

[eternalcuriouslearner]

The OpenAI SDK does already depend on pydantic (>= 1.9.0, < 3), so it will always be present at runtime. However, I'd recommend keeping the explicit pydantic >= 2, < 3 dependency for the following reason:

The instrumentation code in response_extractors.py uses pydantic v2-specific APIs:

ConfigDict (class-based model config)
model_validate() (replaces v1's parse_obj)
model_rebuild() (forward reference resolution)
from_attributes=True config option
The OpenAI SDK's constraint (>= 1.9.0) technically allows pydantic v1, which would cause runtime ImportError/AttributeError failures. Our stricter >= 2 pin acts as an install-time guardrail to prevent that.

In practice, newer OpenAI SDK versions also use pydantic v2 internally, so this is mostly a concern for users with older SDK versions pinned. But removing the explicit dep trades a small cosmetic win for a potential hard-to-debug runtime failure — not worth the risk IMO. Lmk if this is okay with you.