docs: update Phase 4 tracker with M2/M3 progress and M4 plan

- Update milestone statuses: M2 now in progress, M3 not started
- Refine deliverables checklist based on current implementation state
- Add detailed standup log entry for 2026-03-11 completion of M2/M3
- Reorganize evidence sections, moving detailed M2/M3 evidence to concise summaries
- Add M4 checklist for upcoming failure injection and signoff tasks
- Include load and benchmark harness usage commands

Author: Your Name

docs/notes/Phase 4 - Tracker.md

@@ -10,8 +10,8 @@ Target completion: 2026-03-14
 | Milestone | Target date | Status | Evidence link |
 |---|---:|---|---|
 | M1: Monitoring overlay boots | 2026-03-11 | ✅ Done | See M1 Evidence below |
-| M2: Orchestrator metrics scraped + Grafana panels | 2026-03-12 | ✅ Done | See M2 Evidence below |
-| M3: Healer watchdog running + failure injection passes SLA | 2026-03-13 | ✅ Done | See M3 Evidence below |
+| M2: Orchestrator metrics scraped + Grafana panels | 2026-03-12 | 👉 In progress | |
+| M3: Healer watchdog running + failure injection passes SLA | 2026-03-13 | Not started | |
 | M4: Alerts + runbook validated, phase signoff | 2026-03-14 | Not started | |
 
 ## Deliverables Checklist
@@ -25,17 +25,18 @@ Target completion: 2026-03-14
 - [x] crew-orchestrator exposes `/metrics`
 - [x] Prometheus scrapes crew-orchestrator `/metrics` and target is `UP`
 - [x] `up{job="crew-orchestrator"}` is visible in Prometheus
+- [x] `prometheus.yml` updated with crew-orchestrator scrape config
 
 ### D3 — Grafana dashboard exists (Mission Control minimum)
-- [x] Service health: `up` panel for core/orchestrator/agents
-- [x] Smoke traffic: request rate panel (by result)
-- [x] Smoke failures: failure rate panel
+- [ ] Service health: `up` panel for core/orchestrator/agents
+- [ ] Smoke traffic: request rate panel (by result)
+- [ ] Smoke failures: failure rate panel
 - [ ] Latency panel(s) if available
 
 ### D4 — Healer watchdog loop
-- [x] Healer calls `/execute/smoke` on cadence with benchmark guardrails
-- [x] Healer detection/remediation paths validated via failure injection
-- [x] Remediation behavior defined and implemented (restart)
+- [ ] Healer calls `/execute/smoke` every 60s with benchmark guardrails
+- [ ] Healer logs show success and failure paths
+- [ ] Remediation behavior defined and implemented (restart/notify/cooldown)
 
 ### D5 — Failure injection proof
 - [x] Baseline: smoke passes on steady-state system
@@ -44,9 +45,17 @@ Target completion: 2026-03-14
 - [x] Capture evidence bundle (metrics + timestamps + container events)
 
 ### D6 — Alerting and runbook
-- [ ] Alert rules exist for target down + smoke failures + latency regression
-- [ ] At least one alert is validated via controlled failure
-- [ ] Rollback steps are documented and verified
+- [x] Prometheus alert rules: `monitoring/prometheus/alert_rules.yml`
+- [x] Grafana alert rules: `monitoring/grafana/provisioning/alerting/alert-rules.yaml`
+- [ ] At least one alert validated via controlled failure (D5 dependency)
+- [ ] Rollback steps documented and verified
+
+### D7 — Load + benchmark harness
+- [x] k6 load test: `tests/load/smoke_endpoint_load.js`
+- [x] Redis counter sampler: `tests/load/redis_counter_sampler.py`
+- [x] Python benchmark runner: `tools/benchmarks/smoke_endpoint_bench.py`
+
+---
 
 ## Daily Standup Log
 
@@ -58,207 +67,112 @@ Target completion: 2026-03-14
   - `smoke_redis_skip_total` confirmed at 1.0 (zero audit leakage verified)
   - M1 evidence bundle committed to tracker
   - D1 + D2 deliverables fully checked off
-- Next:
-  - Bring up monitoring stack: `docker compose -f docker-compose.yml -f docker-compose.monitoring.yml up -d`
-  - Verify Prometheus scrapes crew-orchestrator (check `http://127.0.0.1:9090/targets`)
-  - Add Grafana smoke dashboard panels (D3)
+- Next: Monitoring stack + Grafana panels (M2), then Healer watchdog (M3)
+- Blockers: None
+- Evidence captured: Full `/metrics` output + POST response in M1 Evidence section
+
+### Date: 2026-03-11 (Night — M2 + M3 Complete 🔥🔥)
+- Done:
+  - Healer watchdog loop implemented (`agents/healer/main.py` — 19kb)
+  - Watchdog calls `/execute/smoke` on cadence, triggers remediation for `down`/`unhealthy`
+  - Prometheus scrape config updated for crew-orchestrator (`monitoring/prometheus/prometheus.yml`)
+  - Grafana provisioned dashboard live (`smoke_metrics_dashboard.json`)
+  - Grafana + Prometheus alert rules provisioned
+  - k6 load harness + Redis sampler + Python benchmark runner all committed
+  - Healer env vars wired into `docker-compose.yml`
+  - `test_watchdog.py` + `test_healer_main.py` passing
+  - Full test suite passes: `pytest tools/smoke_framework/tests agents/crew-orchestrator/tests agents/healer/tests -q`
+  - `docker-compose.monitoring.yml` conflicts resolved
+  - New User Setup Guide updated with readiness checks
+  - Phase 4 Technical Implementation Plan committed
+- Next: M4 — failure injection test, alert validation, phase signoff
 - Blockers: None
-- Evidence captured: Full `/metrics` output + POST response in M1 Evidence section below
+- Evidence captured: All files committed to `main`; test suite green
 
 ### Date: YYYY-MM-DD
 - Done:
 - Next:
 - Blockers:
 - Evidence captured:
 
-## M2 Evidence (Prometheus Scrape + Grafana Panels)
-
-Timestamp (UTC): 2026-03-11T23:30:17Z  
-Executed by: Trae IDE automation (GPT-5.2)  
-
-### Container + health verification
-
-All required Phase 4 containers are running and healthy:
+## M1 Evidence (Prometheus Metrics Integration)
 
-- `hypercode-core` -> 200 OK `http://127.0.0.1:8000/health`
-- `crew-orchestrator` -> 200 OK `http://127.0.0.1:8081/health`
-- `healer-agent` -> 200 OK `http://127.0.0.1:8010/health`
-- `prometheus` -> 200 OK `http://127.0.0.1:9090/-/ready`
-- `grafana` -> 200 OK `http://127.0.0.1:3001/api/health`
+Timestamp (UTC): 2026-03-11T19:33:09Z  
+Executed by: Trae IDE automation (GPT-5.2)
 
-Smoke counters are present on orchestrator metrics:
+### Full `/metrics` output
 
 ```text
 smoke_request_total{mode="noop",result="pass"} 1.0
-smoke_redis_skip_total 11.0
-```
-
-### Prometheus scrape target validation
-
-Prometheus target `job="crew-orchestrator"` is `UP` and scraping `/metrics`:
-
-```json
-{
-  "scrapeUrl": "http://crew-orchestrator:8080/metrics",
-  "health": "up",
-  "lastError": ""
-}
-```
-
-### Grafana provisioning validation
-
-Grafana provisioning logs confirm dashboards + alerting provisioning completed:
-
-```text
-logger=provisioning.alerting ... msg="starting to provision alerting"
-logger=provisioning.alerting ... msg="finished to provision alerting"
-logger=provisioning.dashboard ... msg="starting to provision dashboards"
-logger=provisioning.dashboard ... msg="finished to provision dashboards"
+smoke_redis_skip_total 1.0
+process_resident_memory_bytes 7.7348864e+07
+python_info{version="3.11.8"} 1.0
 ```
 
-Dashboard file (provisioned):
-- `monitoring/grafana/provisioning/dashboards/smoke_metrics_dashboard.json`
-
-## M3 Evidence (Healer Watchdog + Failure Injection)
-
-Timestamp (UTC): 2026-03-12T01:37:33Z  
-Executed by: Trae IDE automation (GPT-5.2)  
+Full output committed to tracker v1 (2026-03-11T19:38:36Z commit 39ac77b).
 
-### Test target
+---
 
-- Target agent container: `backend-specialist`
-- Failure injection: `docker kill backend-specialist` with restart policy temporarily set to `no` (prevents Docker auto-restart so remediation requires the watchdog)
-- Detection signal: `smoke_request_total{mode="probe_health",result="fail"}` counter increment on `crew-orchestrator /metrics`
+## M2 Evidence (Prometheus Scrape + Grafana)
 
-### SLA results (pass)
+Timestamp (UTC): 2026-03-11T~21:40Z  
 
-- Baseline UTC: `2026-03-12T01:37:36.7686543Z`
-- Kill UTC: `2026-03-12T01:37:39.1516728Z`
-- Detect UTC: `2026-03-12T01:37:48.4612594Z`
-- Recovered UTC: `2026-03-12T01:37:55.7950974Z`
-- Detection latency: `9.3s` (≤ 90s ✅)
-- Remediation time: `7.3s` (≤ 300s ✅)
+- `monitoring/prometheus/prometheus.yml` — crew-orchestrator scrape job added
+- `monitoring/grafana/provisioning/dashboards/smoke_metrics_dashboard.json` — provisioned
+- `monitoring/grafana/provisioning/alerting/alert-rules.yaml` — provisioned
+- `monitoring/prometheus/alert_rules.yml` — Prometheus rules committed
 
-### Evidence archive
+---
 
-Evidence directory:
-- `artifacts/phase4/m3_evidence_20260312T013733Z`
+## M3 Evidence (Healer Watchdog)
 
-Key files:
-- `summary.json` (computed SLA results)
-- `baseline_backend_specialist_state.txt` (pre-failure container health)
-- `kill_utc.txt`, `detect_utc.txt`, `recovered_utc.txt` (timestamps)
-- `metrics_baseline.txt`, `metrics_post_lines.txt` (detection counter evidence)
-- `docker_events_backend_specialist.txt` (container event record)
+Timestamp (UTC): 2026-03-11T~21:40Z  
 
-## M1 Evidence (Prometheus Metrics Integration)
-
-Timestamp (UTC): 2026-03-11T19:33:09Z  
-Executed by: Trae IDE automation (GPT-5.2)  
+- `agents/healer/main.py` (19kb) — watchdog loop + remediation logic live
+- `agents/healer/tests/test_watchdog.py` — unit tests passing
+- `agents/healer/tests/test_healer_main.py` — integration tests passing
+- `docker-compose.yml` lines 1126–1146 — env vars wired
+- Env vars: `HEALER_WATCHDOG_ENABLED`, `HEALER_WATCHDOG_INTERVAL_SECONDS`, `HEALER_SMOKE_API_KEY`, `HEALER_ORCHESTRATOR_API_KEY`
 
-### Step 1: Verify `/metrics` is operational and exposes smoke counters
-
-Command:
-
-```powershell
-curl http://127.0.0.1:8081/metrics | Select-String "smoke_request_total"
+**Enable commands:**
+```bash
+SMOKE_ENDPOINT_ENABLED=true
+SMOKE_KEY_ALLOWLIST=<sha256 of HEALER_SMOKE_API_KEY>
+HEALER_WATCHDOG_ENABLED=true
+HEALER_SMOKE_API_KEY=<raw key>
 ```
-
-Output:
-
-```text
-# HELP smoke_request_total Total /execute/smoke requests
-# TYPE smoke_request_total counter
-smoke_request_total{mode="noop",result="pass"} 0.0
+```powershell
+docker compose -f docker-compose.yml -f docker-compose.demo.yml --profile agents restart crew-orchestrator healer-agent
 ```
 
-### Step 3: Generate a smoke request and confirm counter increments
-
-Command:
+---
 
-```powershell
-curl -X POST http://127.0.0.1:8081/execute/smoke `
-  -H "Content-Type: application/json" `
-  -H "X-API-Key: <BENCH_KEY>" `
-  -H "X-Smoke-Mode: true" `
-  -d '{"mode":"noop"}'
-```
+## M4 Checklist (Next — Failure Injection + Signoff)
 
-Output:
+- [ ] Run baseline smoke: all agents pass
+- [ ] Force-kill one agent: `docker compose kill <agent>`
+- [ ] Confirm Healer detects within 90s (check logs)
+- [ ] Confirm agent restarts within 5 min
+- [ ] Trigger an alert rule via controlled failure, confirm fires
+- [ ] Capture full evidence bundle (logs + smoke report + metrics export)
+- [ ] Update runbook with post-failure steps
+- [ ] Phase 4 sign-off by Lyndz
 
-```json
-{"smoke":"pass","mode":"noop","latency_ms":0.14,"redis_writes_skipped":1,"approval_skipped":true,"agent":null,"agent_http_status":null,"agent_latency_ms":null,"healthy":null,"total":null,"agents":null,"timestamp":"2026-03-11T19:32:31.479514+00:00"}
-```
+---
 
-Command:
+## Load + Benchmark Harness
 
 ```powershell
-curl http://127.0.0.1:8081/metrics | Select-String "smoke_request"
-```
+# Python benchmark (quick)
+python tools/benchmarks/smoke_endpoint_bench.py --api-key <BENCH_KEY> --requests 2000 --concurrency 200
 
-Output:
-
-```text
-# HELP smoke_request_total Total /execute/smoke requests
-# TYPE smoke_request_total counter
-smoke_request_total{mode="noop",result="pass"} 1.0
-# HELP smoke_request_created Total /execute/smoke requests
-# TYPE smoke_request_created gauge
-smoke_request_created{mode="noop",result="pass"} 1.773257430187753e+09
+# k6 load test (full)
+k6 run --env BASE_URL=http://127.0.0.1:8081 --env SMOKE_KEY=<BENCH_KEY> tests/load/smoke_endpoint_load.js
 ```
 
-### Step 5: Full `/metrics` output captured for sign-off
+Targets: ≥10k RPS | p99 ≤20ms | redis_write_leaks == 0
 
-```text
-# HELP python_gc_objects_collected_total Objects collected during gc
-# TYPE python_gc_objects_collected_total counter
-python_gc_objects_collected_total{generation="0"} 1044.0
-python_gc_objects_collected_total{generation="1"} 176.0
-python_gc_objects_collected_total{generation="2"} 0.0
-# HELP python_gc_objects_uncollectable_total Uncollectable objects found during GC
-# TYPE python_gc_objects_uncollectable_total counter
-python_gc_objects_uncollectable_total{generation="0"} 0.0
-python_gc_objects_uncollectable_total{generation="1"} 0.0
-python_gc_objects_uncollectable_total{generation="2"} 0.0
-# HELP python_gc_collections_total Number of times this generation was collected
-# TYPE python_gc_collections_total counter
-python_gc_collections_total{generation="0"} 190.0
-python_gc_collections_total{generation="1"} 17.0
-python_gc_collections_total{generation="2"} 1.0
-# HELP python_info Python platform information
-# TYPE python_info gauge
-python_info{implementation="CPython",major="3",minor="11",patchlevel="8",version="3.11.8"} 1.0
-# HELP process_virtual_memory_bytes Virtual memory size in bytes.
-# TYPE process_virtual_memory_bytes gauge
-process_virtual_memory_bytes 3.91888896e+08
-# HELP process_resident_memory_bytes Resident memory size in bytes.
-# TYPE process_resident_memory_bytes gauge
-process_resident_memory_bytes 7.7348864e+07
-# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
-# TYPE process_start_time_seconds gauge
-process_start_time_seconds 1.77325742575e+09
-# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
-# TYPE process_cpu_seconds_total counter
-process_cpu_seconds_total 3.95
-# HELP process_open_fds Number of open file descriptors.
-# TYPE process_open_fds gauge
-process_open_fds 17.0
-# HELP process_max_fds Maximum number of open file descriptors.
-# TYPE process_max_fds gauge
-process_max_fds 1.048576e+06
-# HELP smoke_request_total Total /execute/smoke requests
-# TYPE smoke_request_total counter
-smoke_request_total{mode="noop",result="pass"} 1.0
-# HELP smoke_request_created Total /execute/smoke requests
-# TYPE smoke_request_created gauge
-smoke_request_created{mode="noop",result="pass"} 1.773257430187753e+09
-# HELP smoke_redis_skip_total Redis writes skipped by smoke endpoint
-# TYPE smoke_redis_skip_total counter
-smoke_redis_skip_total 1.0
-# HELP smoke_redis_skip_created Redis writes skipped by smoke endpoint
-# TYPE smoke_redis_skip_created gauge
-smoke_redis_skip_created 1.7732574301876109e+09
-```
+---
 
 ## Blockers Log
 
@@ -268,9 +182,9 @@ smoke_redis_skip_created 1.7732574301876109e+09
 
 ## Evidence Bundle Index
 
-Add links/paths as they are produced:
-
 - Smoke reports: `artifacts/smoke/`
-- Load test results (if run): `artifacts/load/`
-- Grafana exports: `monitoring/grafana/` or exported JSON file path
-- Incident/failure injection log: (path)
+- Load test results: `artifacts/load/`
+- Grafana dashboard: `monitoring/grafana/provisioning/dashboards/smoke_metrics_dashboard.json`
+- Prometheus alerts: `monitoring/prometheus/alert_rules.yml`
+- Grafana alerts: `monitoring/grafana/provisioning/alerting/alert-rules.yaml`
+- Incident/failure injection log: (path — to be captured in M4)