Webhook Delivery Enhancements

[bokelley]

Webhook Delivery Enhancements

Building on the thread-safe webhook delivery service implemented in #316, these enhancements will improve reliability, monitoring, and user experience.

Background

We now have a production-ready webhook delivery service (webhook_delivery_service.py) that is shared by all adapters (mock and GAM). It provides:

• Thread-safe delivery with proper locking
• AdCP V2.3 compliance
• Audit logging
• Admin UI dashboard

Proposed Enhancements

1. Implement GAM Reporting API Integration

Priority: High

The GAM reporting manager currently has a placeholder for fetching delivery metrics. Implement actual GAM Reporting API calls.

File: src/adapters/gam/managers/reporting.py
Method: _fetch_gam_delivery_metrics()

Requirements:

• Use GAM ReportService to fetch line item delivery data
• Fetch: impressions, clicks, spend (CPM/CPC revenue)
• Handle date range queries
• Cache/optimize repeated queries
• Error handling for API limits

Reference: GAM Reporting API Docs

2. Add Webhook Retry Queue

Priority: Medium

Currently, failed webhooks are retried 3 times immediately. Add a persistent retry queue for better reliability.

Features:

• Database-backed retry queue
• Exponential backoff (1m, 5m, 15m, 1h, 6h)
• Max retry limit (configurable per tenant)
• Retry status visible in webhook dashboard
• Manual retry button in admin UI

Tables:

• webhook_delivery_queue - Pending retries
• Add retry_count to audit logs

3. Webhook Health Monitoring Dashboard

Priority: Medium

Enhance the webhooks dashboard with health metrics and alerts.

Features:

• Success rate by principal/media buy
• Average delivery latency
• Failed delivery trends (chart)
• Webhook endpoint health status
• Alert configuration (email/Slack when failure rate > X%)

New Route: /tenant/{tenant_id}/webhooks/health

4. Email/Slack Alerts for Webhook Failures

Priority: Low

Send notifications when webhooks are consistently failing.

Features:

• Configure alert thresholds per tenant
• Alert channels: Email, Slack
• Alert conditions:
  • Failure rate > X% over Y minutes
  • N consecutive failures
  • Endpoint unreachable
• Alert history tracking

Integration:

• Use existing Slack webhook configuration
• Add email alerts via SendGrid/AWS SES

5. Webhook Delivery Metrics & Performance

Priority: Low

Add detailed metrics collection for webhook delivery.

Metrics:

• Delivery latency (p50, p95, p99)
• Success/failure rates
• Endpoint response times
• Payload sizes
• Retry statistics

Storage:

• Time-series data (consider TimescaleDB extension)
• Prometheus metrics export
• Grafana dashboard support

6. Per-Principal Webhook Configuration UI

Priority: Low

Currently webhook configuration is done via database or API. Add UI for managing webhooks.

Features:

• Add/edit/delete webhook endpoints
• Test webhook delivery (send sample payload)
• Enable/disable webhooks
• View webhook history per principal
• Webhook authentication configuration

New Routes:

• /tenant/{tenant_id}/principals/{principal_id}/webhooks
• /tenant/{tenant_id}/principals/{principal_id}/webhooks/test

Implementation Notes

Dependencies

• GAM API client setup required for Welcome to salesagent Discussions! #1
• Consider adding Celery/Redis for retry queue (fix: Configure Conductor workspace with unique ports and fix database indentation #2)
• Metrics may need InfluxDB or Prometheus (feat: AI-Driven Product & Creative Format Management System #5)

Testing

• Each enhancement should include unit tests
• Integration tests for GAM API calls
• Load testing for retry queue
• UI tests for new dashboards

Documentation

• Update docs/webhooks.md with new features
• API documentation for webhook configuration
• Admin guide for monitoring and alerts

Success Criteria

• GAM campaigns report actual delivery data
• Failed webhooks are automatically retried
• Admins can monitor webhook health at a glance
• Teams are alerted to webhook issues before they escalate
• Webhook performance is measured and optimized
• Principals can manage their own webhook configurations

Related

• Add AdCP V2.3 compliant delivery webhook simulation for mock adapter #316 - Original webhook delivery implementation
• src/services/webhook_delivery_service.py - Core service
• src/adapters/gam/managers/reporting.py - GAM reporting
• templates/webhooks.html - Admin UI