Skip to content

Add trust_env#97

Open
AdrianAtZyte wants to merge 2 commits intomainfrom
support-system-proxies
Open

Add trust_env#97
AdrianAtZyte wants to merge 2 commits intomainfrom
support-system-proxies

Conversation

@AdrianAtZyte
Copy link
Copy Markdown

Fixes #96.

@codecov
Copy link
Copy Markdown

codecov bot commented Mar 26, 2026

Codecov Report

❌ Patch coverage is 69.44444% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 98.24%. Comparing base (b84b70c) to head (bad7988).

Files with missing lines Patch % Lines
zyte_api/__main__.py 47.61% 9 Missing and 2 partials ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##              main      #97      +/-   ##
===========================================
- Coverage   100.00%   98.24%   -1.76%     
===========================================
  Files           18       18              
  Lines          716      740      +24     
  Branches        74       77       +3     
===========================================
+ Hits           716      727      +11     
- Misses           0       10      +10     
- Partials         0        3       +3     
Files with missing lines Coverage Δ
zyte_api/_async.py 98.58% <100.00%> (-1.42%) ⬇️
zyte_api/_sync.py 100.00% <ø> (ø)
zyte_api/__main__.py 91.05% <47.61%> (-8.95%) ⬇️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a new trust_env option (and CLI flag) to allow aiohttp requests to honor environment-based proxy settings, addressing proxy-gated environments described in issue #96.

Changes:

  • Add trust_env parameter to AsyncZyteAPI and ZyteAPI, and plumb it into session creation.
  • Add an opt-in --trust-env CLI flag and pass it through to both the client and the underlying aiohttp session.
  • Extend test coverage for trust_env forwarding/behavior and close created sessions to avoid resource leaks.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
zyte_api/_sync.py Adds trust_env to the sync client and forwards it to the async client.
zyte_api/_async.py Stores trust_env on the client and ensures created sessions inherit it.
zyte_api/__main__.py Adds --trust-env flag and passes it into client/session creation; refactors input/output file handling.
tests/test_utils.py Adds coverage for create_session(trust_env=...) and closes sessions.
tests/test_sync.py Verifies sync client forwards trust_env into async client initialization.
tests/test_main.py Verifies CLI run path passes trust_env through and parses --trust-env.
tests/test_auth.py Adjusts subprocess env handling to avoid leaking auth env vars into CLI tests.
tests/test_async.py Verifies async session/get paths inherit client trust_env.
CHANGES.rst Documents the new opt-in trust_env parameter and CLI flag.
Comments suppressed due to low confidence (1)

zyte_api/main.py:142

  • The CLI now treats INPUT as a plain string path and later opens it with Path(...).open(). This changes behavior vs argparse.FileType: passing '-' will no longer read from stdin, and missing/unreadable files will raise an uncaught exception instead of a clean argparse error. Consider restoring FileType (or explicitly supporting '-' -> sys.stdin) and validating/opening failures via parser.error for a user-friendly CLI.
    p.add_argument(
        "INPUT",
        help=(
            "Path to an input file (see 'Command-line client > Input file' in "
            "the docs for details)."
        ),
    )

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@iayanpahwa
Copy link
Copy Markdown

iayanpahwa commented Mar 27, 2026

Thanks for the fix @AdrianAtZyte

Testing Results: Verified in proxy-gated environment

Tested the support-system-proxies branch in an isolated Docker setup where direct DNS is blocked and all traffic must go through an HTTP proxy.

Test Environment

  • Proxy: ubuntu/squid container on a shared Docker network
  • Client: python:3.12 container with --dns 127.0.0.1 (breaks direct DNS)
  • Env vars: HTTPS_PROXY=http://proxy:3128, HTTP_PROXY=http://proxy:3128
  • Install: pip install git+https://github.com/zytedata/python-zyte-api.git@support-system-proxies

Environment Validation

Direct DNS is broken (by design):

$ python3 -c "import socket; socket.getaddrinfo('api.zyte.com', 443)"
DNS broken: [Errno -3] Temporary failure in name resolution

Proxy path works (urllib respects env vars):

$ python3 -c "import urllib.request; urllib.request.urlopen('https://api.zyte.com', timeout=5)"
Proxy works: HTTPError HTTP Error 404: Not Found

Test Results

Without --trust-env (bug — hangs on DNS):

$ zyte-api urls.txt --api-key <key> --output results.jsonl
# Hangs indefinitely — aiohttp attempts direct DNS, never reaches proxy
# Had to Ctrl+C

With --trust-env (fix — connects through proxy):

INFO:zyte_api:
Status codes:
[(200, 1)]

The fix works as expected. --trust-env correctly enables aiohttp to route through the system proxy. The flag is opt-in, so default behavior is unchanged.

Tested on behalf of #96.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

aiohttp ignores HTTP_PROXY/HTTPS_PROXY env vars : CLI failing with DNS errors in proxy-gated environments

4 participants