Skip to content

Tool suggestion: anybrowse MCP for web scraping with Cloudflare bypass #4936

@kc23go

Description

@kc23go

Context

ADK agents doing web research hit Cloudflare-protected sites. The built-in google_search tool finds URLs but agents then fail to actually read the page content on most high-value sites.

Suggestion: anybrowse as a custom ADK tool

anybrowse handles Cloudflare bypass via real residential Chrome and returns clean markdown -- useful as a complement to Google Search in ADK agents.

from google.adk.tools import FunctionTool
import requests

def scrape_url(url: str) -> str:
    """Scrape a URL and return clean markdown content, including Cloudflare-protected sites."""
    r = requests.post("https://anybrowse.dev/scrape", json={"url": url})
    if r.ok:
        return r.json().get("markdown", "")
    return f"Scrape failed: HTTP {r.status_code}"

scrape_tool = FunctionTool(func=scrape_url)

Or as an MCP server:

{
  "mcpServers": {
    "anybrowse": {"type": "streamable-http", "url": "https://anybrowse.dev/mcp"}
  }
}

Metadata

Metadata

Assignees

No one assigned

    Labels

    mcp[Component] Issues about MCP supportspam[Status] Issues suspected of having comments which are spamtools[Component] This issue is related to tools

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions