Skip to content

Document LLM Observability LLM provider integration endpoints #3557

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
api-clients-generation-pipeline merged 1 commit into from
May 21, 2026
Merged

Document LLM Observability LLM provider integration endpoints #3557

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
870 changes: 870 additions & 0 deletions .generator/schemas/v2/openapi.yaml
View file
Open in desktop

Large diffs are not rendered by default.

189 changes: 189 additions & 0 deletions docs/datadog_api_client.v2.model.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18358,6 +18358,48 @@ datadog\_api\_client.v2.model.llm\_obs\_annotation\_schema module
:members:
:show-inheritance:

datadog\_api\_client.v2.model.llm\_obs\_anthropic\_effort module
----------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.llm_obs_anthropic_effort
:members:
:show-inheritance:

datadog\_api\_client.v2.model.llm\_obs\_anthropic\_metadata module
------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.llm_obs_anthropic_metadata
:members:
:show-inheritance:

datadog\_api\_client.v2.model.llm\_obs\_anthropic\_thinking\_config module
--------------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.llm_obs_anthropic_thinking_config
:members:
:show-inheritance:

datadog\_api\_client.v2.model.llm\_obs\_anthropic\_thinking\_type module
------------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.llm_obs_anthropic_thinking_type
:members:
:show-inheritance:

datadog\_api\_client.v2.model.llm\_obs\_azure\_open\_ai\_metadata module
------------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.llm_obs_azure_open_ai_metadata
:members:
:show-inheritance:

datadog\_api\_client.v2.model.llm\_obs\_bedrock\_metadata module
----------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.llm_obs_bedrock_metadata
:members:
:show-inheritance:

datadog\_api\_client.v2.model.llm\_obs\_content\_block module
-------------------------------------------------------------

Expand Down Expand Up @@ -19345,6 +19387,125 @@ datadog\_api\_client.v2.model.llm\_obs\_experiments\_response module
:members:
:show-inheritance:

datadog\_api\_client.v2.model.llm\_obs\_inference\_code module
--------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.llm_obs_inference_code
:members:
:show-inheritance:

datadog\_api\_client.v2.model.llm\_obs\_inference\_content module
-----------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.llm_obs_inference_content
:members:
:show-inheritance:

datadog\_api\_client.v2.model.llm\_obs\_inference\_content\_value module
------------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.llm_obs_inference_content_value
:members:
:show-inheritance:

datadog\_api\_client.v2.model.llm\_obs\_inference\_error\_response module
-------------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.llm_obs_inference_error_response
:members:
:show-inheritance:

datadog\_api\_client.v2.model.llm\_obs\_inference\_function module
------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.llm_obs_inference_function
:members:
:show-inheritance:

datadog\_api\_client.v2.model.llm\_obs\_inference\_message module
-----------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.llm_obs_inference_message
:members:
:show-inheritance:

datadog\_api\_client.v2.model.llm\_obs\_inference\_run\_result module
---------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.llm_obs_inference_run_result
:members:
:show-inheritance:

datadog\_api\_client.v2.model.llm\_obs\_inference\_tool module
--------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.llm_obs_inference_tool
:members:
:show-inheritance:

datadog\_api\_client.v2.model.llm\_obs\_inference\_tool\_call module
--------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.llm_obs_inference_tool_call
:members:
:show-inheritance:

datadog\_api\_client.v2.model.llm\_obs\_inference\_tool\_result module
----------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.llm_obs_inference_tool_result
:members:
:show-inheritance:

datadog\_api\_client.v2.model.llm\_obs\_integration\_account module
-------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.llm_obs_integration_account
:members:
:show-inheritance:

datadog\_api\_client.v2.model.llm\_obs\_integration\_inference\_request module
------------------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.llm_obs_integration_inference_request
:members:
:show-inheritance:

datadog\_api\_client.v2.model.llm\_obs\_integration\_inference\_response module
-------------------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.llm_obs_integration_inference_response
:members:
:show-inheritance:

datadog\_api\_client.v2.model.llm\_obs\_integration\_model module
-----------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.llm_obs_integration_model
:members:
:show-inheritance:

datadog\_api\_client.v2.model.llm\_obs\_integration\_model\_region\_prefix\_overrides module
--------------------------------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.llm_obs_integration_model_region_prefix_overrides
:members:
:show-inheritance:

datadog\_api\_client.v2.model.llm\_obs\_integration\_name module
----------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.llm_obs_integration_name
:members:
:show-inheritance:

datadog\_api\_client.v2.model.llm\_obs\_internal\_reasoning module
------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.llm_obs_internal_reasoning
:members:
:show-inheritance:

datadog\_api\_client.v2.model.llm\_obs\_label\_schema module
------------------------------------------------------------

Expand Down Expand Up @@ -19373,6 +19534,27 @@ datadog\_api\_client.v2.model.llm\_obs\_metric\_score\_type module
:members:
:show-inheritance:

datadog\_api\_client.v2.model.llm\_obs\_open\_ai\_metadata module
-----------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.llm_obs_open_ai_metadata
:members:
:show-inheritance:

datadog\_api\_client.v2.model.llm\_obs\_open\_ai\_reasoning\_effort module
--------------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.llm_obs_open_ai_reasoning_effort
:members:
:show-inheritance:

datadog\_api\_client.v2.model.llm\_obs\_open\_ai\_reasoning\_summary module
---------------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.llm_obs_open_ai_reasoning_summary
:members:
:show-inheritance:

datadog\_api\_client.v2.model.llm\_obs\_project\_data\_attributes\_request module
---------------------------------------------------------------------------------

Expand Down Expand Up @@ -19625,6 +19807,13 @@ datadog\_api\_client.v2.model.llm\_obs\_trace\_interaction\_type module
:members:
:show-inheritance:

datadog\_api\_client.v2.model.llm\_obs\_vertex\_ai\_metadata module
-------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.llm_obs_vertex_ai_metadata
:members:
:show-inheritance:

datadog\_api\_client.v2.model.log module
----------------------------------------

Expand Down
126 changes: 126 additions & 0 deletions examples/v2/llm-observability/CreateLLMObsIntegrationInference.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
"""
Run an LLM inference returns "OK" response
"""

from datadog_api_client import ApiClient, Configuration
from datadog_api_client.v2.api.llm_observability_api import LLMObservabilityApi
from datadog_api_client.v2.model.llm_obs_anthropic_effort import LLMObsAnthropicEffort
from datadog_api_client.v2.model.llm_obs_anthropic_metadata import LLMObsAnthropicMetadata
from datadog_api_client.v2.model.llm_obs_anthropic_thinking_config import LLMObsAnthropicThinkingConfig
from datadog_api_client.v2.model.llm_obs_anthropic_thinking_type import LLMObsAnthropicThinkingType
from datadog_api_client.v2.model.llm_obs_azure_open_ai_metadata import LLMObsAzureOpenAIMetadata
from datadog_api_client.v2.model.llm_obs_bedrock_metadata import LLMObsBedrockMetadata
from datadog_api_client.v2.model.llm_obs_inference_content import LLMObsInferenceContent
from datadog_api_client.v2.model.llm_obs_inference_content_value import LLMObsInferenceContentValue
from datadog_api_client.v2.model.llm_obs_inference_function import LLMObsInferenceFunction
from datadog_api_client.v2.model.llm_obs_inference_message import LLMObsInferenceMessage
from datadog_api_client.v2.model.llm_obs_inference_tool import LLMObsInferenceTool
from datadog_api_client.v2.model.llm_obs_inference_tool_call import LLMObsInferenceToolCall
from datadog_api_client.v2.model.llm_obs_inference_tool_result import LLMObsInferenceToolResult
from datadog_api_client.v2.model.llm_obs_integration_inference_request import LLMObsIntegrationInferenceRequest
from datadog_api_client.v2.model.llm_obs_integration_name import LLMObsIntegrationName
from datadog_api_client.v2.model.llm_obs_open_ai_metadata import LLMObsOpenAIMetadata
from datadog_api_client.v2.model.llm_obs_open_ai_reasoning_effort import LLMObsOpenAIReasoningEffort
from datadog_api_client.v2.model.llm_obs_open_ai_reasoning_summary import LLMObsOpenAIReasoningSummary
from datadog_api_client.v2.model.llm_obs_vertex_ai_metadata import LLMObsVertexAIMetadata

body = LLMObsIntegrationInferenceRequest(
anthropic_metadata=LLMObsAnthropicMetadata(
effort=LLMObsAnthropicEffort.MEDIUM,
thinking=LLMObsAnthropicThinkingConfig(
budget_tokens=1024,
type=LLMObsAnthropicThinkingType.ENABLED,
),
),
azure_openai_metadata=LLMObsAzureOpenAIMetadata(
deployment_id="my-gpt4-deployment",
model_version="0613",
resource_name="my-azure-resource",
),
bedrock_metadata=LLMObsBedrockMetadata(
region="us-east-1",
),
frequency_penalty=0.0,
json_schema='{"type":"object","properties":{"answer":{"type":"string"}}}',
max_completion_tokens=1024,
max_tokens=1024,
messages=[
LLMObsInferenceMessage(
content="What is the capital of France?",
contents=[
LLMObsInferenceContent(
type="text",
value=LLMObsInferenceContentValue(
text="Hello, how can I help you?",
tool_call=LLMObsInferenceToolCall(
arguments=dict([("location", "San Francisco")]),
name="get_weather",
tool_id="call_abc123",
type="function",
),
tool_call_result=LLMObsInferenceToolResult(
name="get_weather",
result="The weather in San Francisco is 68°F and sunny.",
tool_id="call_abc123",
type="function",
),
),
),
],
id="msg_001",
role="user",
tool_calls=[
LLMObsInferenceToolCall(
arguments=dict([("location", "San Francisco")]),
name="get_weather",
tool_id="call_abc123",
type="function",
),
],
tool_results=[
LLMObsInferenceToolResult(
name="get_weather",
result="The weather in San Francisco is 68°F and sunny.",
tool_id="call_abc123",
type="function",
),
],
),
],
model_id="gpt-4o",
openai_metadata=LLMObsOpenAIMetadata(
reasoning_effort=LLMObsOpenAIReasoningEffort.MEDIUM,
reasoning_summary=LLMObsOpenAIReasoningSummary.AUTO,
),
presence_penalty=0.0,
temperature=0.7,
tools=[
LLMObsInferenceTool(
function=LLMObsInferenceFunction(
description="Get the current weather for a location.",
name="get_weather",
parameters=dict([("properties", "{'location': {'type': 'string'}}"), ("type", "object")]),
),
type="function",
),
],
top_k=50,
top_p=1.0,
vertex_ai_metadata=LLMObsVertexAIMetadata(
location="us-central1",
project="my-gcp-project",
project_ids=[
"my-gcp-project",
],
),
)

configuration = Configuration()
configuration.unstable_operations["create_llm_obs_integration_inference"] = True
with ApiClient(configuration) as api_client:
api_instance = LLMObservabilityApi(api_client)
response = api_instance.create_llm_obs_integration_inference(
integration=LLMObsIntegrationName.OPENAI, account_id="account_id", body=body
)

print(response)
17 changes: 17 additions & 0 deletions examples/v2/llm-observability/ListLLMObsIntegrationAccounts.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
"""
List LLM integration accounts returns "OK" response
"""

from datadog_api_client import ApiClient, Configuration
from datadog_api_client.v2.api.llm_observability_api import LLMObservabilityApi
from datadog_api_client.v2.model.llm_obs_integration_name import LLMObsIntegrationName

configuration = Configuration()
configuration.unstable_operations["list_llm_obs_integration_accounts"] = True
with ApiClient(configuration) as api_client:
api_instance = LLMObservabilityApi(api_client)
response = api_instance.list_llm_obs_integration_accounts(
integration=LLMObsIntegrationName.OPENAI,
)

print(response)
18 changes: 18 additions & 0 deletions examples/v2/llm-observability/ListLLMObsIntegrationModels.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
"""
List LLM integration models returns "OK" response
"""

from datadog_api_client import ApiClient, Configuration
from datadog_api_client.v2.api.llm_observability_api import LLMObservabilityApi
from datadog_api_client.v2.model.llm_obs_integration_name import LLMObsIntegrationName

configuration = Configuration()
configuration.unstable_operations["list_llm_obs_integration_models"] = True
with ApiClient(configuration) as api_client:
api_instance = LLMObservabilityApi(api_client)
response = api_instance.list_llm_obs_integration_models(
integration=LLMObsIntegrationName.OPENAI,
account_id="account_id",
)

print(response)
3 changes: 3 additions & 0 deletions src/datadog_api_client/configuration.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -272,6 +272,7 @@ def __init__(
"v2.create_llm_obs_dataset_records": False,
"v2.create_llm_obs_experiment": False,
"v2.create_llm_obs_experiment_events": False,
"v2.create_llm_obs_integration_inference": False,
"v2.create_llm_obs_project": False,
"v2.delete_llm_obs_annotation_queue": False,
"v2.delete_llm_obs_annotation_queue_interactions": False,
Expand All @@ -289,6 +290,8 @@ def __init__(
"v2.list_llm_obs_datasets": False,
"v2.list_llm_obs_experiment_events": False,
"v2.list_llm_obs_experiments": False,
"v2.list_llm_obs_integration_accounts": False,
"v2.list_llm_obs_integration_models": False,
"v2.list_llm_obs_projects": False,
"v2.list_llm_obs_spans": False,
"v2.search_llm_obs_experimentation": False,
Expand Down
Loading
Loading