Implement Fetch Web Tool for HTTP content retrieval

[Copilot]

Adds model-agnostic HTTP/HTTPS fetching capability to complement Google Search tool (ADR 0007). Enables direct URL content access with format processing.

Implementation

Core functionality (tools/web/fetch.go, 428 lines):

• HTTP client with configurable timeout (30s default, 5min max) and size limits (1MB default, 50MB max)
• Content processing: text (HTML stripping), JSON (formatting), HTML (structured parsing), raw
• Security: URL scheme validation, size enforcement, timeout protection
• Error classification with machine-readable codes

Tool registration:

• Category: CategorySearchDiscovery, priority 1 (after Google Search)
• Auto-registered via init(), follows existing tool patterns
• Works with all LLM providers (Gemini, OpenAI, Ollama)

Testing (fetch_test.go, 433 lines):

• 22 tests: basic operations, error handling, timeouts, redirects, format processing
• Uses httptest for HTTP mocking
• 100% pass rate

Usage

// Fetch and format JSON API response
format := "json"
input := FetchWebInput{
    URL: "https://api.example.com/data",
    Format: &format,
    Headers: map[string]string{"Authorization": "Bearer token"},
}
output := FetchWebHandler(ctx, input)

Integration

Workflow pattern with Google Search:

1. Google Search finds relevant URLs
2. Fetch Web retrieves content from specific URLs
3. Agent synthesizes answer

Updated exports in tools/tools.go. Documentation in CHANGELOG.md, README.md (tool count: 21→22), and ADR 0007.

CodeQL: 0 vulnerabilities

Original prompt

[Chronological Review: The conversation began with the user outlining the need for a new tool to fetch web content, detailing the context and problem statement. The user then presented a decision-making process that included various options for implementation, ultimately choosing to create a custom HTTP fetch tool. The user provided a comprehensive implementation strategy, technical details, and a directory structure for the new tool. The conversation culminated in the user requesting a summary of the current state of the ADR document they were working on, specifically focusing on the Fetch Web Tool implementation.]

[Intent Mapping: The user explicitly requested to document the Fetch Web Tool implementation, including its context, decision drivers, options considered, and the chosen implementation strategy. They also sought to outline the technical details, architecture integration, and usage examples, culminating in a detailed implementation checklist.]

[Technical Inventory: Technologies mentioned include Go programming language, HTTP client libraries, JSON and HTML parsing, and error handling strategies. The user also discussed architectural patterns for tool registration and integration within the existing adk-code framework.]

[Code Archaeology: The user detailed the structure of the new tool, including files like fetch.go, fetch_test.go, and init.go, along with the definitions for input and output types, handler functions, and helper functions for processing content.]

[Progress Assessment: The user has completed the initial draft of the ADR document, outlining the Fetch Web Tool's purpose, decision-making process, and implementation strategy. They are now in the phase of finalizing the document and preparing for implementation.]

[Context Validation: All critical information regarding the Fetch Web Tool's implementation, including its purpose, decision outcomes, and technical details, has been captured for continuation.]

[Recent Commands Analysis: The last agent command involved the user requesting a summary of the ADR document they were working on. The tool results included a comprehensive outline of the Fetch Web Tool's implementation, decision-making process, and technical specifications. The agent was actively working on finalizing the ADR document when the token budget was exceeded, focusing on the Fetch Web Tool's implementation details and usage examples.]

1. Conversation Overview: - Primary Objectives: The user aimed to document the Fetch Web Tool implementation, stating, "Adding HTTP web content fetching capability to adk-code agent" and detailing its necessity for fetching documentation and real-time data. - Session Context: The conversation flowed from identifying the need for a new tool, exploring implementation options, and detailing the chosen approach, culminating in a request for a summary of the ADR document. - User Intent Evolution: The user's needs evolved from outlining the problem to detailing the implementation strategy and seeking a summary of the current state of the ADR document.

2. Technical Foundation:

• Go: The primary programming language used for the implementation of the Fetch Web Tool.
• HTTP Client: Utilized for making requests to fetch web content.
• JSON/HTML Parsing: Techniques discussed for processing fetched content.

3. Codebase Status:

• fetch.go:
  • Purpose: Implements the Fetch Web Tool logic.
  • Current State: Drafted with input/output types and handler functions.
  • Key Code Segments: Includes FetchWebInput and FetchWebOutput structures, and the FetchWebHandler function.
  • Dependencies: Relies on Go's standard library for HTTP requests and JSON/HTML processing.

4. Problem Resolution:

• Issues Encountered: Need for a tool to fetch web content directly.
• Solutions Implemented: Decision to create a custom HTTP fetch tool.
• Debugging Context: No specific debugging issues reported; focus on implementation.
• Lessons Learned: Importance of having a model-agnostic tool for flexibility.

5. Progress Tracking:

• Completed Tasks: Drafting of the ADR document and outlining the Fetch Web Tool's purpose and implementation.
• Partially Complete Work: Finalizing the ADR document and preparing for implementation.
• Validated Outcomes: Clear understanding of the tool's requirements and implementation strategy.

6. Active Work State:

• Current Focus: Finalizing the ADR document for the Fetch Web Tool.
• Recent Context: The user was detailing the implementation strategy and usage examples when the summary request was made.
• Working Code: Code snippets for the Fetch Web Tool's input/output structures and handler functions were discussed.
• Immediate Context: The user was preparing to finalize the ADR document.

7. Recent Operations:

• Last Agent Commands: The user requested a summary of the ADR document they were working on.
• Tool Results Summary: The results included a comprehensive outline of the Fetch Web Tool's...

Created from VS Code via the GitHub Pull Request extension.

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.