Feature branch for httpx migration by mwdd146980 · Pull Request #22676 · DataDog/integrations-core

mwdd146980 · 2026-02-18T18:55:54Z

What does this PR do?

Introduces library-agnostic HTTP infrastructure and decouples integration tests from requests, laying the groundwork for the requests → httpx migration across ~76 integrations.

Motivation

Migrating from requests to httpx requires decoupling tests from requests internals first — otherwise regressions are undetectable. This PR establishes that boundary. See the RFC for full context.

Approach

Phase 1: Protocol definitions — HTTPClientProtocol, HTTPResponseProtocol, a library-agnostic exception hierarchy, and MockHTTPResponse (response object independent of requests.Response). Re-exports in http.py for convenient imports.

Step 1 (PR #22710, merged) — mock_http pytest fixture (auto-loaded via ddev plugin) that patches AgentCheck.http at the property level. The OM V2 scraper was changed to reuse check.http, so a single patch covers both. 9 integrations migrated.

Step 2 (PR #22722, merged) — Added get_header()/set_header() (case-insensitive) to RequestsWrapper and HTTPClientProtocol. Migrated 19 config assertion tests from patching requests.Session to asserting against check.http.options and check.http.get_header(). Production code in vault updated to use set_header(). Mock fixture extended with client.options as a real dict.

28 integrations now use the library-agnostic test layer on this branch.

See: Test Decoupling Plan, Step 1, Step 2

Status

Work in progress. This PR is the feature branch — subsequent steps are stacked PRs that merge here after review.

Step	PR	Status
Phase 1: Protocol + MockHTTPResponse	#22676, #22680	✅ On feature branch
Step 1: mock_http fixture + 9 integrations	#22710	✅ Merged
Step 2: Config assertions + get/set_header (19 intg)	#22722	✅ Merged
Step 3a: Widen except clauses + align MockHTTPResponse	#22864	🔄 In review
Step 3b: Swap mock_http_response fixture	#22935	🔄 In review

Verification

All existing unit tests pass with the new fixtures
MockHTTPResponse covered by dedicated tests
Formatting validated with ddev test -fs

Review checklist (to be filled by reviewers)

Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
Add the qa/skip-qa label if the PR doesn't need to be tested during QA.
If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

codecov · 2026-02-18T19:26:56Z

Codecov Report

❌ Patch coverage is 94.09190% with 27 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.35%. Comparing base (31eda08) to head (a05a40a).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

datadog_checks_base/changelog.d/22676.added

datadog_checks_base/datadog_checks/base/utils/http_protocol.py

datadog_checks_base/datadog_checks/base/utils/http_exceptions.py

mwdd146980 · 2026-02-20T23:48:21Z

Feature branch for httpx migration #22676 : 4 dependent PRs (#22704 , #22710 , #22722 and 1 other) 👈 (View in Graphite)
master

This stack of pull requests is managed by Graphite. Learn more about stacking.

@NouemanKHAL

* Add HTTP protocol definitions for httpx migration Introduce HTTPClientProtocol and HTTPResponseProtocol as library-agnostic interfaces, enabling test decoupling before migrating from requests to httpx. These protocols define the contract that both RequestsWrapper and the future HTTPXClient must implement. There should be zero runtime overhead and no impact on existing code. * Add changelog entry * Add pragma no cover to protocol stubs for coverage Protocol method stubs with ellipsis are compile-time type hints that are never executed. Add pragma no cover comments to indicate these lines are intentionally not covered by runtime tests. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Add close() method to HTTPResponseProtocol Production code calls response.close() in finally blocks to release connections back to the pool (e.g., openmetrics/mixins.py:549, openmetrics/v2/scraper/base_scraper.py:447, prometheus/mixins.py:388). Without close() in the protocol, new HTTP client implementations could pass protocol checks but raise AttributeError when these code paths execute, or leak connections. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Apply suggestions from code review Co-authored-by: NouemanKHAL <noueman.khalikine@datadoghq.com> * Apply suggestion from @NouemanKHAL Co-authored-by: NouemanKHAL <noueman.khalikine@datadoghq.com> * Apply suggestion from @NouemanKHAL, delete this file Co-authored-by: NouemanKHAL <noueman.khalikine@datadoghq.com> * Remove protocol interface tests and fix type annotation Delete test_http_protocol.py as testing protocol interfaces provides no value - we would only be testing Python's structural subtyping internals rather than actual implementation behavior. Also fix typo in HTTPResponseProtocol.__enter__ return type annotation where ResponseProtocol was referenced instead of HTTPResponseProtocol. * Run code formatter (remove trailing newline) --------- Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com> Co-authored-by: NouemanKHAL <noueman.khalikine@datadoghq.com>

Introduces HTTPError, HTTPRequestError, HTTPStatusError, HTTPTimeoutError, HTTPConnectionError, and HTTPSSLError as abstract, library-agnostic exception types. These provide a stable interface for the httpx migration — checks that handle HTTP errors can catch these types without caring whether the underlying client is requests or httpx. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

@overload

…de behavior Both methods return bytes when decode_unicode=False and str when decode_unicode=True. Iterator[bytes] was wrong for the decode_unicode=True case (e.g. the openmetrics mixin calls iter_lines(decode_unicode=True) and processes results as strings). The precise fix would use @overload with Literal[True]/Literal[False] but that adds verbosity with no practical benefit at this stage. Iterator[bytes | str] is the honest union — correct for both runtime cases. Also aligns delimiter type to bytes | str | None, matching http_testing.py. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Add MockHTTPResponse for library-agnostic HTTP response mocking MockHTTPResponse implements HTTPResponseProtocol without depending on requests or httpx. It supports the full response API (.json(), .text, .content, .status_code, .headers, .cookies, .elapsed), streaming via iter_content() and iter_lines(), raise_for_status() using HTTPStatusError, and context manager usage. Demonstrates usage by migrating test_authtoken.py from MockResponse to MockHTTPResponse — a drop-in replacement with protocol compliance. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Expand MockHTTPResponse usage to test_openmetrics.py and test_kerberos_unit.py Also adds three missing API surface members discovered during migration: - encoding attribute (accessed by openmetrics mixin before iter_lines) - close() no-op (called by openmetrics mixin after response processing) - Removes _stream_consumed guard so the same instance can be reused across repeated mock.MagicMock(return_value=...) calls Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Reformat with ddev test --fmt * Fix headers mutation and add case-insensitive header dict Two correctness fixes in MockHTTPResponse: 1. When json_data and headers are both provided, the caller's headers dict was mutated in-place via setdefault(). Now copies before modifying. 2. self.headers was a plain dict (case-sensitive). HTTP headers are case-insensitive per RFC 7230 §3.2. Replace with _CaseInsensitiveDict that stores keys lowercased, matching requests.Response behaviour. --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

No other file in the codebase uses module-level docstrings. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Adds convenience re-exports so integrations can import HTTP exceptions from the existing datadog_checks.base.utils.http import path rather than a new module, supporting a phased migration to httpx. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…sponse Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Redesign mock_http fixture to patch RequestsWrapper at class level Patches get/post/put/delete/head/patch on RequestsWrapper so all three HTTP paths are intercepted: AgentCheck.http, OpenMetrics V2 scraper, and kube* health check handlers. Real RequestsWrapper instances are still created, so check.http.options remains accessible. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Migrate falco tests from requests.Session.get patch to mock_http fixture Replace direct mock.patch('requests.Session.get') with the library-agnostic mock_http fixture and MockHTTPResponse from http_testing.py. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Migrate strimzi tests from requests.Session.get patch to mock_http fixture Replace direct mocker.patch('requests.Session.get') with the library-agnostic mock_http fixture and MockHTTPResponse from http_testing.py. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix E402: move pytest_plugins after all imports Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Migrate couchbase tests from requests.Session.get patch to mock_http fixture Replace direct mocker.patch('requests.Session.get') with the library-agnostic mock_http fixture and MockHTTPResponse from http_testing.py. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Rename mock_http fixture to mock_http_client The fixture mocks the HTTP client (RequestsWrapper), not responses. mock_http_client is more precise and avoids confusion with the existing mock_http_response fixture from datadog_checks.dev. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Migrate ray tests from requests.Session.get patch to mock_http_client fixture Replace direct mocker.patch('requests.Session.get') with the library-agnostic mock_http_client fixture and MockHTTPResponse from http_testing.py. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Migrate tekton tests from requests.Session.get patch to mock_http_client fixture Replace direct mocker.patch('requests.Session.get') with the library-agnostic mock_http_client fixture and MockHTTPResponse from http_testing.py. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Switch from pytest_plugins to direct import for fixture registration Use direct import with noqa: F401 to match the existing codebase pattern. No other integration uses pytest_plugins for fixture registration. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Revert rename: mock_http_client → mock_http The fixture name mock_http_client was added in the previous commit, but the team prefers the shorter mock_http name. Revert all usages across integrations and the fixture definition in http_testing.py. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Redesign mock_http: use create_autospec(HTTPClientProtocol) + PropertyMock on AgentCheck.http - mock_http now patches AgentCheck.http via PropertyMock with a create_autospec(HTTPClientProtocol) client, constraining the mock to the protocol interface and enforcing call signatures - AgentCheck.http return type updated from RequestsWrapper to HTTPClientProtocol - OM V2 scraper reuses check.http directly instead of constructing its own RequestsWrapper; options access guarded with hasattr for mock compatibility Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Shorten docstring * Migrate kubevirt_api, kubevirt_controller, kubevirt_handler tests to mock_http fixture Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Fix Accept header mutation side-effect in OM V2 scraper base_scraper.py shares check.http via `self.http = check.http`. The previous code mutated `self.http.options['headers']['Accept']` which bled through to check.http since they are the same object, breaking tests that assert check.http.options['headers']['Accept'] == '*/*'. Fix: store the accept header as self._accept_header and inject it per-request via extra_headers in send_request. Using extra_headers (not headers=) is required because RequestsWrapper._request uses ChainMap — passing headers= directly would shadow the entire session headers dict rather than merging a single key. Update test assertions in three files to check scraper._accept_header instead of scraper.http.options['headers']['Accept']. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Fix bentoml tests broken by scraper sharing check.http After self.http = check.http in the OM V2 scraper, patching BentomlCheck.http at the class level intercepted scraper HTTP calls too, causing AttributeError on the minimal mock response. Replace both patch('BentomlCheck.http') blocks with URL-based dispatch on the already-patched requests.Session.get mock. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Fix Vault auth headers lost when OM V2 scraper reuses check.http Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Raise NotImplementedError for options_method in mock_http fixture Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Move mock_http fixture to datadog_checks_dev pytest plugin Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Remove call_count assertion from bentoml test Asserting on the number of HTTP calls ties the test to implementation details rather than observable behavior (metrics, service checks). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Drop unused get_mock variable in bentoml test Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- base_scraper: use self.check.http instead of local check.http parameter - base_scraper: move accept_header from instance variable to local in send_request - openmetrics tests: replace scraper._accept_header assertion with call_args header inspection - bentoml tests: remove duplicate assert_all_metrics_covered call - bentoml tests: replace requests.HTTPError with HTTPStatusError from http_exceptions Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…_accept_header Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…rim comments Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Add get_header/set_header (case-insensitive) to RequestsWrapper and HTTPClientProtocol, then migrate 16 integration test suites from patching requests.Session internals to asserting against check.http.options and using the mock_http fixture. - Add get_header/set_header to RequestsWrapper with case-insensitive key lookup; use set_header in Vault's __init__ auth header setup - Extend HTTPClientProtocol with options property and get/set_header - Migrate config assertion tests: airflow, consul, couch, druid, ecs_fargate, envoy, etcd, gitlab_runner, marathon, mesos_master, mesos_slave, nginx, openmetrics, php_fpm, squid, teamcity - Update mock_http fixture to expose client.options as a real dict - Flatten test classes to top-level functions in test_headers.py - Remove dead assertions in rabbitmq and torchserve tests

temporal-github-worker-1 bot added agent/review-requested ecosystems/review-requested product/review-requested labels Feb 18, 2026

datadog-agent-integrations-bot bot added the base_package label Feb 18, 2026

NouemanKHAL reviewed Feb 20, 2026

View reviewed changes

datadog_checks_base/changelog.d/22676.added Outdated Show resolved Hide resolved

NouemanKHAL reviewed Feb 20, 2026

View reviewed changes

datadog_checks_base/datadog_checks/base/utils/http_protocol.py Show resolved Hide resolved

NouemanKHAL reviewed Feb 20, 2026

View reviewed changes

datadog_checks_base/datadog_checks/base/utils/http_exceptions.py Outdated Show resolved Hide resolved

mwdd146980 force-pushed the mwdd146980/httpx-migration-base branch from beb72ad to 07a9003 Compare February 20, 2026 19:16

mwdd146980 mentioned this pull request Feb 20, 2026

Phase 2: HTTPXWrapper initial implementation #22704

Draft

3 tasks

This was referenced Feb 20, 2026

Forward instance config (auth, TLS, timeout, proxy, headers) to HTTPXWrapper #22705

Draft

Add Kerberos and NTLM auth adapters for HTTPXWrapper #22706

Draft

mwdd146980 force-pushed the mwdd146980/httpx-migration-base branch from 97854d1 to 8fc9e4f Compare February 23, 2026 14:24

This was referenced Feb 23, 2026

Add mock_http fixture; migrate 9 intg to mock_http #22710

Merged

Migrate config assertion tests away from requests #22722

Merged

mwdd146980 force-pushed the mwdd146980/httpx-migration-base branch 2 times, most recently from 102bf30 to 6f90874 Compare February 27, 2026 06:04

datadog-agent-integrations-bot bot added integration/php_fpm integration/rabbitmq integration/squid integration/teamcity integration/druid integration/airflow integration/torchserve labels Mar 19, 2026

mwdd146980 and others added 22 commits March 20, 2026 13:56

Update PR changelog

06d798c

Rename changelog

cc4b634

Address Noueman's comments

21ad097

Remove docstring on http_exceptions.py

2d36811

Update changelog: consolidate into 22676.added, drop 22677.added

31a6555

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Remove module docstrings from http_testing.py and test_http_testing.py

7e54dab

No other file in the codebase uses module-level docstrings. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Use unittest.mock.MagicMock for raw socket mock in MockHTTPResponse

7108b1e

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Replace _CaseInsensitiveDict with plain lowercased dict in MockHTTPRe…

13bb344

…sponse Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Fix ruff lint: remove unnecessary parens around single-line string

07b98aa

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add changelog entries for datadog_checks_base and datadog_checks_dev

5131417

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Fix TestUseLatestSpec: assert on _use_latest_spec instead of removed …

28fec1f

…_accept_header Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add changelog entry for vault intg

60ba647

Revise message for vault intg changelog entry

8a7830c

Clean up MockHTTPResponse: top-level imports, flatten test classes, t…

67a1872

…rim comments Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

mwdd146980 force-pushed the mwdd146980/httpx-migration-base branch from 932c02a to a05a40a Compare March 20, 2026 18:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature branch for httpx migration#22676

Feature branch for httpx migration#22676
mwdd146980 wants to merge 22 commits intomasterfrom
mwdd146980/httpx-migration-base

mwdd146980 commented Feb 18, 2026 •

edited

Loading

Uh oh!

codecov bot commented Feb 18, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mwdd146980 commented Feb 20, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

mwdd146980 commented Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Approach

Status

Verification

Review checklist (to be filled by reviewers)

Uh oh!

codecov bot commented Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mwdd146980 commented Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mwdd146980 commented Feb 18, 2026 •

edited

Loading

codecov bot commented Feb 18, 2026 •

edited

Loading

mwdd146980 commented Feb 20, 2026 •

edited

Loading