Skip to content

Commit 092324b

Browse files
rodrigobr-msftCopilot
rodrigobr-msft
and
Copilot
authored
Data-driven integration tests (#249)
* basic integration utility * Integration test suite factory implementation * Implementing core models for integration testing * AutoClient mockup * Adding runner starter code * Adding foundational classes * Drafting AgentClient and ResponseClient implementations * Spec test * Filling in more implementation details * More files * Cleaning up implementations * Adding expect replies sending method * Beginning unit tests * Adding integration decor from sample test cases * Integration from service url tests * _handle_conversation implementation for response_client * AgentClient tests completed * Creating response client tests * Hosting server for response client * Response client tests completed * Beginning refactor of aiohttp runner * Unit test updates * Fixing issues in refactor * Fixed TestIntegrationFromSample unit test * Another commit * Reorganizing files * Completed TestIntegrationFromServiceUrl unit tests * Reformatting with black * Quickstart integration test beginning * Draft of quickstart sample setup * Environment config connection * Renaming messaging endpoint to agent url * Removing unnecessary files * Fixing agent client payload sending * First successful integration test * Beginning foundational test cases * TypingIndicator test * Adding more test cases * More foundational integration test cases * Reorganizing testing tools into package * Polished the testing framework package * Adding verbose logging for results with benchmark tool * General cleanup * Adding README * Revising README * Formatting * Addressing PR comments * Removing unused import * Update dev/microsoft-agents-testing/microsoft_agents/testing/sdk_config.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update dev/microsoft-agents-testing/pyproject.toml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update dev/microsoft-agents-testing/microsoft_agents/testing/auth/generate_token.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update dev/microsoft-agents-testing/microsoft_agents/testing/integration/core/environment.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Removing unused code * Adding field assertion helpers * Implementing field assertion types * Adding data driven test foundation * Separating assertions from checkers * Adjusting return values for check_activity with refactor * Fixing package name in pyproject.toml * Assertion test cases * _parse_assertion unit tests * More assertion tests and fixes to data normalization logic * Fixing more test cases with newest refactor * assert_activity tests * Reorganizing tests * Copyright comments * DDT implementation outline * DataDrivenTest class * select_activity implementation for assertions * inverted assertions for quantifiers * ActivityAssertion class implementation * DataDrivenTest implementation * Populate function for dictionaries * Populate tests * Selector class implementation * Assertion tests completed for now * Adding ActivityAssertion tests * DataDrivenTest implementation connected with default config * More test cases and formatting * Data driven testing single test setup support * Formatting * Dynamic data driven test loading * Another commit * DataDrivenTest class implementation with accompanying unit tests * Recursive test loading with parent links * Slight reorg * Formatting * Adding DataDrivenTest cases * Fixing test cases * Test case assertions * Reorganizing integration tests * Removed unused files * Reverting payload for benchmark * Added working integration tests * Cleanup for PR * Fixing typing annotation on ddt decorator * Updated README.md * Adding Quickstart integration tests * Fixing integration test name resolution * Adding initial for claire-bot tests * Claire-bot tests * Adding data driven tests for claire-bot * Reformatting with black * Update dev/integration/agents/claire_bot/python/src/agent.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update dev/integration/agents/claire_bot/python/src/agent.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * PR comments * Formatting * Minor fixes --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
1 parent de0083f commit 092324b

146 files changed

Lines changed: 9117 additions & 144 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎dev/README.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
# Development Tools
22

3+
DISCLAIMER: the content of this directory is experimental and not meant for production use.
4+
35
Development utilities for the Microsoft Agents for Python project.
46

57
## Contents
@@ -11,7 +13,7 @@ Development utilities for the Microsoft Agents for Python project.
1113
## Quick Setup
1214

1315
```bash
14-
./install.sh
16+
pip install -e ./microsoft-agents-testing/ --config-settings editable_mode=compat
1517
```
1618

1719
## Benchmarking

‎dev/install.sh‎

Lines changed: 0 additions & 1 deletion
This file was deleted.

dev/microsoft-agents-testing/tests/manual_test/__init__.py renamed to dev/integration/__init__.py

File renamed without changes.

‎dev/integration/agents/__init__.py‎

Whitespace-only changes.

‎dev/integration/agents/basic_agent/__init__.py‎

Whitespace-only changes.

‎dev/integration/agents/basic_agent/python/README.md‎

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
# 🤖 Agents SDK Test Framework's Python Bot
2+
3+
This Python bot is part of the Agents SDK Test Framework. It exercises agent behaviors, validates responses, and helps iterate on integrations with LLMs and tools.
4+
5+
## Highlights ✨
6+
- ⚙️ Test-runner for validating agent flows and tool/function calling
7+
- 🧠 Integrates with LLM providers (Azure OpenAI, Semantic Kernel)
8+
- 🖥️ Uses Microsoft Agents SDK packages for hosting and activity management
9+
10+
## 🚀 Getting Started
11+
12+
### 🛠️ Prerequisites
13+
- Python 3.9+
14+
- `pip` (Python package manager)
15+
16+
### 📦 Installation
17+
1. Install dependencies:
18+
```powershell
19+
pip install --pre --no-deps -r pre_requirements.txt
20+
pip install -r requirements.txt
21+
```
22+
#### ℹ️ Why are there two installation steps?
23+
24+
**Dependency installation is split into two steps to ensure reliability and avoid conflicts:**
25+
26+
- **Step 1:** `pre_requirements.txt` — Installs core Microsoft Agents SDK packages. These may require pre-release flags or special handling, and installing them first (without dependency resolution) helps prevent version clashes.
27+
- **Step 2:** `requirements.txt` — Installs the rest of the project dependencies, after the core packages are in place, to ensure compatibility and a smooth setup.
28+
29+
This approach helps avoid dependency issues and guarantees all required packages are installed in the correct order.
30+
31+
### ⚙️ Set up Environment Variables
32+
Copy or rename `.envLocal` to `.env` and fill in the required values (keys, endpoints, etc.).
33+
34+
> 💡 Tip: The repo often uses Azure resources (Azure OpenAI / Bot Service) in examples.
35+
36+
### ▶️ Running the Agent
37+
Start the agent locally:
38+
```powershell
39+
python app.py
40+
```
41+
42+
## 📁 Project Layout
43+
```
44+
Agent/python/
45+
agent.py
46+
app.py
47+
config.py
48+
requirements.txt
49+
requirements2.txt
50+
pre_requirements.txt
51+
.env
52+
.envLocal
53+
weather/
54+
agents/
55+
weather_forecast_agent.py
56+
weather_forecast_agent_response.py
57+
plugins/
58+
adaptive_card_plugin.py
59+
date_time_plugin.py
60+
weather_forecast_plugin.py
61+
weather_forecast.py
62+
```
63+
64+
This launches the process that hosts the agent and exposes the `/api/messages` endpoint.
65+
66+
## 📚 Key Dependencies
67+
- `microsoft-agents-hosting-core`, `microsoft-agents-hosting-aiohttp`, `microsoft-agents-activity`, `microsoft-agents-authentication-msal` — Microsoft Agents SDK packages
68+
- `semantic-kernel` — LLM orchestration
69+
- `openai` — Azure OpenAI integration
70+
71+
## Health & Messaging Endpoints
72+
- Health check: (if exposed) `GET /` should return 200
73+
- Messaging / activity endpoint: `POST /api/messages` (see `app.py`)
74+
75+
## Agent Flow 🔁
76+
1. The test runner accepts scenario inputs (natural language user messages).
77+
2. It forwards activity payloads to the agent runtime.
78+
3. The agent may call functions/tools (e.g., weather, date/time).
79+
4. The runner validates the agent's JSON / Adaptive Card outputs and records results.
80+
81+
## Contributing
82+
- Open a PR with changes and add a short description of the test scenarios you added or modified.

‎dev/integration/agents/basic_agent/python/__init__.py‎

Whitespace-only changes.

‎dev/integration/agents/basic_agent/python/env.TEMPLATE‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
CONNECTIONS__SERVICE_CONNECTION__SETTINGS__CLIENTID=
2+
CONNECTIONS__SERVICE_CONNECTION__SETTINGS__CLIENTSECRET=
3+
CONNECTIONS__SERVICE_CONNECTION__SETTINGS__TENANTID=
4+
5+
AZURE_OPENAI_ENDPOINT=
6+
AZURE_OPENAI_API_KEY=
7+
AZURE_OPENAI_API_VERSION=
8+
AZURE_OPENAI_DEPLOYMENT_NAME=

‎dev/integration/agents/basic_agent/python/pre_requirements.txt‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
microsoft-agents-hosting-core
2+
microsoft-agents-hosting-aiohttp
3+
microsoft-agents-authentication-msal
4+
microsoft-agents-activity
5+
microsoft-agents-hosting-teams
6+
microsoft-agents-copilotstudio-client
7+
microsoft-agents-storage-blob
8+
microsoft-agents-storage-cosmos

‎dev/integration/agents/basic_agent/python/requirements.txt‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
openai
2+
openai-agents
3+
semantic-kernel
4+
microsoft-agents-hosting-aiohttp
5+
microsoft-agents-authentication-msal
6+
microsoft-agents-hosting-teams
7+
microsoft-agents-copilotstudio-client
8+
microsoft-agents-storage-blob
9+
microsoft-agents-storage-cosmos

0 commit comments

Comments
 (0)