Python: HandoffBuilder incompatible with Azure AI Agent Service (tools cannot be added at request time)

[frdeange]

Summary

HandoffBuilder does not work with agents created via AzureAIProjectAgentProvider because Azure AI Responses API requires tools to be defined at agent creation time, not at request time. HandoffBuilder internally adds handoff tools to cloned agents via _apply_auto_tools(), but these tools are never registered in Azure AI Agent Service.

Additionally, if users pre-create handoff tools to work around this limitation, _apply_auto_tools() raises a ValueError for duplicate tool names, preventing the workaround.

---

Environment

• Package: agent-framework (Python)
• Provider: AzureAIProjectAgentProvider (uses Azure AI Foundry Responses API)
• Python: 3.13+

---

Problem Description

Root Cause

The comment in _project_provider.py (lines 231-232) explicitly states:

"This is required because Azure AI Responses API doesn't accept tools at request time"

When HandoffBuilder.build() runs:

1. _clone_chat_agent() creates a copy of the local ChatAgent wrapper
2. _apply_auto_tools() adds handoff tools to cloned_agent.default_options["tools"]
3. The Azure AI agent definition is NOT updated - it still has no handoff tools

When the workflow runs, AzureAIClient._prepare_options() strips tools from request options because Azure AI doesn't accept them at request time. The LLM never sees the handoff tools.

The ValueError Problem

If users try to pre-create handoff tools at agent creation time:

@tool(name="handoff_to_AbsenceAgent", description="Transfer to AbsenceAgent")
def handoff_to_absence(context: str | None = None) -> str:
    return "Handoff to AbsenceAgent"

router = await provider.create_agent(
    name="RouterAgent",
    tools=[handoff_to_absence],  # Pre-create handoff tool
)

Then _apply_auto_tools() raises:

ValueError: Agent 'RouterAgent' already has a tool named 'handoff_to_AbsenceAgent'. 
Handoff tool name 'handoff_to_AbsenceAgent' conflicts with existing tool.

---

Proposed Solution

1. Add create_handoff_tools() helper function

A public helper that creates handoff tools with the exact signature expected by _AutoHandoffMiddleware:

def create_handoff_tools(
    target_agent_ids: Sequence[str],
    descriptions: Mapping[str, str] | None = None,
) -> list[FunctionTool]:
    """Create handoff tools for pre-registration with Azure AI Agent Service.
    
    Use this when creating agents with AzureAIProjectAgentProvider, where tools
    must be defined at agent creation time rather than at request time.
    
    Args:
        target_agent_ids: List of target agent IDs to create handoff tools for.
        descriptions: Optional mapping of agent ID to custom description.
        
    Returns:
        List of FunctionTool objects ready to pass to create_agent().
        
    Example:
        handoff_tools = create_handoff_tools(["AbsenceAgent", "OvertimeAgent"])
        router = await provider.create_agent(
            name="RouterAgent",
            tools=handoff_tools + other_tools,
        )
    """

2. Modify _apply_auto_tools() to skip existing tools

Change the current behavior from raising ValueError to skipping with a debug log:

# Current (problematic):
if handoff_tool.name in existing_names:
    raise ValueError(...)

# Proposed (compatible):
if handoff_tool.name in existing_names:
    logger.debug(
        f"Handoff tool '{handoff_tool.name}' already exists on agent "
        f"'{resolve_agent_id(agent)}'. Skipping auto-creation."
    )
    continue

This follows the existing pattern used in:

• _merge_options() in _agents.py
• load_tools() in _mcp.py
• merge_tools() in _tooling.py

---

Why This Is Not a Breaking Change

Scenario	Before	After
Local agents (no pre-created tools)	✅ Works	✅ Works (unchanged)
Azure AI (no pre-created tools)	❌ Fails silently	❌ Same (user didn't use helper)
Azure AI (with pre-created tools)	❌ ValueError crash	✅ Now works
Mixed providers	❌ Depends	✅ Works

The change relaxes a restriction rather than adding one. Code that worked before continues to work.

---

Reproduction Steps

from agent_framework import HandoffBuilder
from agent_framework.azure import AzureAIProjectAgentProvider

async def main():
    provider = AzureAIProjectAgentProvider(credential=credential)
    
    async with provider:
        # Create agents in Azure AI Agent Service
        router = await provider.create_agent(
            name="RouterAgent",
            instructions="Route to AbsenceAgent for time-off requests"
        )
        
        absence = await provider.create_agent(
            name="AbsenceAgent", 
            instructions="Handle absence requests"
        )
        
        # Build handoff workflow
        workflow = (
            HandoffBuilder(participants=[router, absence])
            .with_start_agent(router)
            .add_handoff(router, [absence])
            .build()
        )
        
        # Run - handoff fails because RouterAgent doesn't have handoff tools in Azure
        async for event in workflow.run_stream("I want to request time off"):
            print(event)

# Result: RouterAgent talks about transferring but never actually calls handoff tool
# LLM may "hallucinate" tool calls in text output

---

Impact

This issue makes HandoffBuilder unusable with Azure AI Agent Service, which is a significant limitation for enterprise scenarios where:

1. Agents need to be managed/monitored in Azure AI Foundry portal
2. Compliance requires agent definitions to be stored in Azure
3. MCP tools need service-side execution via HostedMCPTool
4. Multi-agent handoff patterns are required

---

Implementation Offer

I would like to contribute a PR implementing this fix. The changes would include:

• New create_handoff_tools() helper function in _handoff.py
• Modified _apply_auto_tools() to skip duplicates with debug log
• Export create_handoff_tools in __init__.py
• Unit tests for the new helper and skip-duplicate behavior

Please let me know if this approach is acceptable or if you'd prefer a different solution.

[moonbox3]

@frdeange, we're fully aware of this. We need the Agent Service to support dynamic tools at runtime for this pattern to work properly.

[moonbox3]

@TaoChenOSU, note: this used to work in this way (being able to toggle auto_apply_tools()) before the large orchestration refactor. FYI.

[frdeange]

@markwallace-microsoft , @moonbox3 . Happy to help with that!!!! I did this approach in PR #3719 but of course do not aware about refactor.

I´m FTE --> frdeange_microsoft, so happy to collaborate if it make sense!!!

Thanks!