Bug Description
The Webshare.list_proxies() method ignores the page parameter in search_params and always fetches ALL pages from the API, causing extremely long execution times (35-70 minutes) instead of fetching just the specified page.
Expected Behavior
When search_params={"page": 1, "page_size": 25} is provided, the method should fetch only page 1 with 25 proxies and return immediately (~1 second).
Actual Behavior
The method loops through ALL pages (2,151 pages for 215,084 proxies) regardless of the page parameter in search_params, taking 35-70 minutes to complete.
Steps to Reproduce
on
from proxyproviders import Webshare
proxy_provider = Webshare(
api_key="your-api-key",
search_params={"mode": "backbone", "page": 1, "page_size": 25}
)
This should fetch only page 1, but instead fetches all 2,151 pages
proxies = proxy_provider.list_proxies(force_refresh=True)## Root Cause
In providers/webshare.py, the _fetch_proxies() method has a while True loop that:
-
Starts with page = 0 and increments page += 1 in each iteration
-
Overrides the page parameter from search_params with the loop counter
-
Continues looping until data.get("next") is None (all pages fetched)
hon
def _fetch_proxies(self) -> List[Proxy]:
all_proxies = []
page = 0
while True: # ← Always loops through all pages
page += 1 # ← Overrides search_params["page"]
default_params = {
"mode": "direct",
"page": page, # ← Ignores search_params["page"]
"page_size": 100,
}
params = {**default_params, **self.search_params}
# ... fetch page ...
if data.get("next") is None:
break # ← Only stops when no more pages## Workaround
Direct API call works correctly:
import requests
response = requests.get(
"https://proxy.webshare.io/api/v2/proxy/list",
headers={"Authorization": f"Token {api_key}"},
params={"mode": "backbone", "page": 1, "page_size": 25}
)
proxies = response.json()["results"] # Returns only page 1## Environment
Impact
- High: Makes the library unusable for fetching a specific page
- Performance: 35-70 minutes vs expected 1 second
- User experience: Appears to hang/freeze
Please fix this problem
Bug Description
The
Webshare.list_proxies()method ignores thepageparameter insearch_paramsand always fetches ALL pages from the API, causing extremely long execution times (35-70 minutes) instead of fetching just the specified page.Expected Behavior
When
search_params={"page": 1, "page_size": 25}is provided, the method should fetch only page 1 with 25 proxies and return immediately (~1 second).Actual Behavior
The method loops through ALL pages (2,151 pages for 215,084 proxies) regardless of the
pageparameter insearch_params, taking 35-70 minutes to complete.Steps to Reproduce
on
from proxyproviders import Webshare
proxy_provider = Webshare(
api_key="your-api-key",
search_params={"mode": "backbone", "page": 1, "page_size": 25}
)
This should fetch only page 1, but instead fetches all 2,151 pages
proxies = proxy_provider.list_proxies(force_refresh=True)## Root Cause
In
providers/webshare.py, the_fetch_proxies()method has awhile Trueloop that:Starts with
page = 0and incrementspage += 1in each iterationOverrides the
pageparameter fromsearch_paramswith the loop counterContinues looping until
data.get("next") is None(all pages fetched)hon
def _fetch_proxies(self) -> List[Proxy]:
all_proxies = []
page = 0
while True: # ← Always loops through all pages
page += 1 # ← Overrides search_params["page"]
default_params = {
"mode": "direct",
"page": page, # ← Ignores search_params["page"]
"page_size": 100,
}
params = {**default_params, **self.search_params}
# ... fetch page ...
if data.get("next") is None:
break # ← Only stops when no more pages## Workaround
Direct API call works correctly:
import requests
response = requests.get(
"https://proxy.webshare.io/api/v2/proxy/list",
headers={"Authorization": f"Token {api_key}"},
params={"mode": "backbone", "page": 1, "page_size": 25}
)
proxies = response.json()["results"] # Returns only page 1## Environment
Impact
Please fix this problem