Strengthen reviewer demo coherence

.gitignore

@@ -2,6 +2,7 @@ __pycache__/
 *.py[cod]
 *.egg-info/
 .pytest_cache/
+.pytest-artifacts*/
 .coverage
 .venv/
 venv/

CONTRIBUTING.md

@@ -17,7 +17,7 @@ Thanks for the interest. This is a solo-maintainer portfolio repository, so the
 - run:
 
   ```bash
-  python -m pip install -e .
+  python -m pip install -e ".[dev]"
   pytest
   ```
 

README.md

@@ -2,7 +2,7 @@
 
 [![CI](https://github.com/stacknil/telemetry-lab/actions/workflows/ci.yml/badge.svg)](https://github.com/stacknil/telemetry-lab/actions/workflows/ci.yml)
 
-Small portfolio prototypes for telemetry analytics, monitoring, and detection-oriented signal processing.
+A local, file-based detection workflow lab for reviewer-verifiable telemetry and detection demos.
 
 Latest milestone: [v0.6.0 — fourth demo and config-change investigation](https://github.com/stacknil/telemetry-lab/releases/latest).
 
@@ -22,7 +22,7 @@ Latest milestone: [v0.6.0 — fourth demo and config-change investigation](https
 
 ## What This Repo Is
 
-`telemetry-lab` is a small portfolio repository for telemetry analytics and constrained detection-oriented workflows. It is organized as four local, file-based demos that are reproducible from committed sample data and intentionally scoped for public review rather than production use.
+`telemetry-lab` is a small portfolio repository for constrained detection workflows. It is not a SIEM, dashboard, or monitoring platform; it is organized as four local, file-based demos that are reproducible from committed sample data and intentionally scoped for public review rather than production use.
 
 ### telemetry-window-demo
 
@@ -47,6 +47,9 @@ python -m pip install -e .
 python -m telemetry_window_demo.cli run --config configs/default.yaml
 ```
 
+Use the same Python interpreter for install, tests, and demo commands. On machines with multiple Python installs, replace `python` with the intended interpreter path.
+To run the test suite in a fresh environment, install the dev extra with `python -m pip install -e ".[dev]"`.
+
 Other demo entrypoints:
 
 - `python -m telemetry_window_demo.cli run-ai-demo`
@@ -56,9 +59,10 @@ Other demo entrypoints:
 Useful inspection commands:
 
 - `python -m telemetry_window_demo.cli summarize --input data/raw/sample_events.jsonl`
-- `python -m telemetry_window_demo.cli summarize --input events.csv --timestamp-col event_time`
 
-That command reads `data/raw/sample_events.jsonl` and regenerates:
+For CSV inputs, pass a `.csv` file to `--input`; use `--timestamp-col` when the timestamp column is not named `timestamp`.
+
+The `run --config configs/default.yaml` command reads `data/raw/sample_events.jsonl` and regenerates:
 
 - `data/processed/features.csv`
 - `data/processed/alerts.csv`
@@ -73,17 +77,26 @@ With the bundled default sample, the current repo state produces:
 - `24` windows
 - `12` alerts after a `60` second cooldown
 
-Why it is worth a quick look:
-
-- it shows a full telemetry path from raw events to operator-facing outputs
-- the sample inputs and outputs are reproducible in-repo
-- a second bundled scenario gives a slightly richer walkthrough without changing the basic CLI flow
-
-![Default alert timeline](data/processed/alerts_timeline.png)
-
-## Demo Variants
-
-Default sample:
+Why it is worth a quick look:
+
+- it shows a full telemetry path from raw events to operator-facing outputs
+- the sample inputs and outputs are reproducible in-repo
+- a second bundled scenario gives a slightly richer walkthrough without changing the basic CLI flow
+
+![Default alert timeline](data/processed/alerts_timeline.png)
+
+## Reviewer Path
+
+For a quick coherence pass across the demos:
+
+1. Run `python -m telemetry_window_demo.cli run --config configs/default.yaml` and confirm `data/processed/summary.json` reports `41` events, `24` windows, and `12` alerts.
+2. Run `python -m telemetry_window_demo.cli run-rule-dedup-demo` and confirm `demos/rule-evaluation-and-dedup-demo/artifacts/dedup_report.md` shows `10` raw hits reduced to `6` retained alerts with `4` suppressions.
+3. Run `python -m telemetry_window_demo.cli run-config-change-demo` and confirm `demos/config-change-investigation-demo/artifacts/investigation_report.md` shows `4` normalized changes, `3` risky changes, and `3` investigations.
+4. Run `python -m telemetry_window_demo.cli run-ai-demo` and confirm `demos/ai-assisted-detection-demo/artifacts/case_report.md` shows `3` deterministic cases with human verification and no final incident verdict.
+
+## Demo Variants
+
+Default sample:
 
 - config: [`configs/default.yaml`](configs/default.yaml)
 - input: `data/raw/sample_events.jsonl`
@@ -129,6 +142,7 @@ Cooldown behavior:
 
 - [`demos/rule-evaluation-and-dedup-demo/README.md`](demos/rule-evaluation-and-dedup-demo/README.md) explains the third demo and links its committed before/after dedup artifacts
 - [`demos/config-change-investigation-demo/README.md`](demos/config-change-investigation-demo/README.md) explains the config-change investigation demo and its committed artifacts
+- [`docs/reviewer-path.md`](docs/reviewer-path.md) maps common review questions to the right demo and artifacts
 - [`docs/sample-output.md`](docs/sample-output.md) summarizes the committed sample artifacts
 - [`docs/roadmap.md`](docs/roadmap.md) sketches the next demo directions
 - [`data/processed/summary.json`](data/processed/summary.json) captures the default run in machine-readable form

demos/ai-assisted-detection-demo/README.md

@@ -65,23 +65,32 @@ Generated artifacts are written to `demos/ai-assisted-detection-demo/artifacts/`
 - deterministic rules: `config/rules.yaml`
 - structured output schema: `config/llm_case_output_schema.json`
 
-## Expected artifacts
-
-- `artifacts/rule_hits.json`
-- `artifacts/case_bundles.json`
-- `artifacts/case_summaries.json`
-- `artifacts/case_report.md`
-- `artifacts/audit_traces.jsonl`
-
-The bundled sample data is designed to produce at least three generated cases.
-
-## Artifact semantics
+## Expected artifacts
+
+- `artifacts/rule_hits.json`
+- `artifacts/case_bundles.json`
+- `artifacts/case_summaries.json`
+- `artifacts/case_report.md`
+- `artifacts/audit_traces.jsonl`
+
+## Expected run summary
+
+The bundled sample run should report:
+
+- `15` raw events normalized into `15` internal events
+- `5` deterministic rule hits
+- `3` grouped cases
+- `3` accepted JSON summaries
+- `0` rejected summaries in the default accepted path
+- `3` audit records
+
+## Artifact semantics
 
 - `rule_hits.json`: deterministic rule hits with rule metadata, ATT&CK mapping, entities, and evidence highlights
 - `case_bundles.json`: grouped cases with severity, rule hits, ATT&CK mappings, raw evidence, and untrusted-data marking
 - `case_summaries.json`: only accepted JSON summaries that passed schema and semantic validation
-- `case_report.md`: analyst-facing report that shows accepted summaries and explicitly notes rejected case summaries
-- `case_report.md`: includes a top-level run integrity section that surfaces rule/config degradation
+- `case_report.md`: analyst-facing report with run counts, accepted summaries, and explicit notes for rejected case summaries
+- `case_report.md`: includes a top-level run integrity section that surfaces rule/config degradation
 - `audit_traces.jsonl`: stable per-record audit log for accepted and rejected paths, using `schema_version = ai-assisted-detection-audit/v1` and including `ts`, `case_id`, `validation_status`, `rejection_reason`, `rule_ids`, `prompt_input_digest`, `evidence_digest`, and bounded response excerpts
 
 ## Rejection behavior
@@ -103,15 +112,27 @@ Use the default sample run artifacts in `artifacts/case_summaries.json`, `artifa
 
 Verify that `CASE-001` appears in all three places, that the `case_id` matches exactly, that `human_verification` is `required`, and that the audit record shows `validation_status = accepted` with `schema_version = ai-assisted-detection-audit/v1`.
 
-### Rejected summary path
-
-Run `pytest tests/test_ai_assisted_detection_demo.py -k "audit_traces_capture_accepted_and_rejected_paths or case_id_mismatch"` and inspect the `case_report.md`, `case_summaries.json`, and `audit_traces.jsonl` artifacts written by the test.
+### Rejected summary path
+
+Run:
+
+```bash
+pytest tests/test_ai_assisted_detection_demo.py -k "audit_traces_capture_accepted_and_rejected_paths or case_id_mismatch" --basetemp .pytest-artifacts-ai-demo-rejections
+```
+
+Then inspect the `case_report.md`, `case_summaries.json`, and `audit_traces.jsonl` files under `.pytest-artifacts-ai-demo-rejections/test_*/artifacts/`.
 
 Verify that the rejected case is absent from `case_summaries.json`, appears in `case_report.md` as `Summary status: rejected`, and has an audit record with `validation_status = rejected` plus a concrete `rejection_reason` such as `missing_required_fields`, `semantic_validation_failed`, or `case_id_mismatch`.
 
-### Degraded coverage path
-
-Run `pytest tests/test_ai_assisted_detection_demo.py -k malformed_attack_metadata_is_rejected_and_recorded` and inspect the generated `case_report.md` and `audit_traces.jsonl`.
+### Degraded coverage path
+
+Run:
+
+```bash
+pytest tests/test_ai_assisted_detection_demo.py -k malformed_attack_metadata_is_rejected_and_recorded --basetemp .pytest-artifacts-ai-demo-degraded
+```
+
+Then inspect the generated `case_report.md` and `audit_traces.jsonl` files under `.pytest-artifacts-ai-demo-degraded/test_*/artifacts/`.
 
 Verify that `case_report.md` exposes `## Run Integrity`, `coverage_degraded: yes`, and the rejected rule id, and that `audit_traces.jsonl` contains a global rejection record with `case_id = null` and `rejection_reason = rule_metadata_validation_failed`.
 

demos/ai-assisted-detection-demo/artifacts/case_report.md

@@ -1,80 +1,90 @@
-# AI-Assisted Detection Demo Report
-
-This report is analyst-facing draft output from a constrained case summarization pipeline.
-Detections and grouping are deterministic. The LLM is limited to structured summarization only.
-Human verification is required. No automated response actions or final incident verdicts are produced.
-
-## Run Integrity
-
-- accepted_rules: AUTH-001, AUTH-002, PROC-001, WEB-001
-- rejected_rules: none
-- coverage_degraded: no
-- rejection_reasons: none
-
-## CASE-001
-
-- Severity: high
-- First seen: 2026-03-27T09:01:55Z
-- Last seen: 2026-03-27T09:02:20Z
-- Rule hits: repeated_failed_logins, successful_login_after_failures
-- ATT&CK: T1110, T1078
-
-Summary: CASE-001 contains 2 deterministic rule hits covering repeated_failed_logins, successful_login_after_failures for principal ops_admin; src_ip 198.51.100.24; host vpn-gw-01 during 2026-03-27T09:01:55Z to 2026-03-27T09:02:20Z. The case warrants analyst review but does not imply a final incident decision.
-
-Likely causes:
-- Repeated password guessing or credential stuffing against the targeted account.
-- A valid credential may have been used after several failed login attempts.
-
-Uncertainty notes:
-- Telemetry is limited to the bundled sample evidence and does not confirm operator intent.
-- The case summary is advisory only and requires human review before any incident classification.
-
-Suggested next steps:
-- Review the raw evidence and confirm whether the activity aligns with an approved administrative task.
-- Check authentication context for MFA state, prior successful logins, and expected source locations.
-- Document the analyst conclusion separately after human verification; do not treat this summary as a final verdict.
-
-## CASE-002
-
-- Severity: medium
-- First seen: 2026-03-27T09:11:10Z
-- Last seen: 2026-03-27T09:11:10Z
-- Rule hits: sensitive_path_scan
-- ATT&CK: T1595
-
-Summary: CASE-002 contains 1 deterministic rule hits covering sensitive_path_scan for src_ip 203.0.113.77; host portal-01 during 2026-03-27T09:11:10Z to 2026-03-27T09:11:10Z. The case warrants analyst review but does not imply a final incident decision.
-
-Likely causes:
-- The source IP appears to be probing sensitive web paths on the exposed application.
-
-Uncertainty notes:
-- Telemetry is limited to the bundled sample evidence and does not confirm operator intent.
-- The case summary is advisory only and requires human review before any incident classification.
-- Prompt-like text appeared in telemetry and was treated strictly as untrusted evidence.
-
-Suggested next steps:
-- Review the raw evidence and confirm whether the activity aligns with an approved administrative task.
-- Compare the web requests with reverse-proxy and WAF logs to determine whether the probing continued.
-- Document the analyst conclusion separately after human verification; do not treat this summary as a final verdict.
-
-## CASE-003
-
-- Severity: high
-- First seen: 2026-03-27T09:20:00Z
-- Last seen: 2026-03-27T09:20:20Z
-- Rule hits: encoded_powershell_execution, encoded_powershell_execution
-- ATT&CK: T1059.001
-
-Summary: CASE-003 contains 2 deterministic rule hits covering encoded_powershell_execution for principal lab_user; host wkstn-07 during 2026-03-27T09:20:00Z to 2026-03-27T09:20:20Z. The case warrants analyst review but does not imply a final incident decision.
-
-Likely causes:
-- Obfuscated PowerShell execution may reflect manual tradecraft or an unsafe script.
-
-Uncertainty notes:
-- Telemetry is limited to the bundled sample evidence and does not confirm operator intent.
-- The case summary is advisory only and requires human review before any incident classification.
-
-Suggested next steps:
-- Review the raw evidence and confirm whether the activity aligns with an approved administrative task.
-- Inspect the originating host timeline and validate whether the encoded PowerShell command matches known tooling.
-- Document the analyst conclusion separately after human verification; do not treat this summary as a final verdict.
+# AI-Assisted Detection Demo Report
+
+This report is analyst-facing draft output from a constrained case summarization pipeline.
+Detections and grouping are deterministic. The LLM is limited to structured summarization only.
+Human verification is required. No automated response actions or final incident verdicts are produced.
+
+## Run Summary
+
+- raw_events: 15
+- normalized_events: 15
+- rule_hits: 5
+- cases: 3
+- accepted_summaries: 3
+- rejected_summaries: 0
+- audit_records: 3
+
+## Run Integrity
+
+- accepted_rules: AUTH-001, AUTH-002, PROC-001, WEB-001
+- rejected_rules: none
+- coverage_degraded: no
+- rejection_reasons: none
+
+## CASE-001
+
+- Severity: high
+- First seen: 2026-03-27T09:01:55Z
+- Last seen: 2026-03-27T09:02:20Z
+- Rule hits: repeated_failed_logins, successful_login_after_failures
+- ATT&CK: T1110, T1078
+
+Summary: CASE-001 contains 2 deterministic rule hits covering repeated_failed_logins, successful_login_after_failures for principal ops_admin; src_ip 198.51.100.24; host vpn-gw-01 during 2026-03-27T09:01:55Z to 2026-03-27T09:02:20Z. The case warrants analyst review but does not imply a final incident decision.
+
+Likely causes:
+- Repeated password guessing or credential stuffing against the targeted account.
+- A valid credential may have been used after several failed login attempts.
+
+Uncertainty notes:
+- Telemetry is limited to the bundled sample evidence and does not confirm operator intent.
+- The case summary is advisory only and requires human review before any incident classification.
+
+Suggested next steps:
+- Review the raw evidence and confirm whether the activity aligns with an approved administrative task.
+- Check authentication context for MFA state, prior successful logins, and expected source locations.
+- Document the analyst conclusion separately after human verification; do not treat this summary as a final verdict.
+
+## CASE-002
+
+- Severity: medium
+- First seen: 2026-03-27T09:11:10Z
+- Last seen: 2026-03-27T09:11:10Z
+- Rule hits: sensitive_path_scan
+- ATT&CK: T1595
+
+Summary: CASE-002 contains 1 deterministic rule hits covering sensitive_path_scan for src_ip 203.0.113.77; host portal-01 during 2026-03-27T09:11:10Z to 2026-03-27T09:11:10Z. The case warrants analyst review but does not imply a final incident decision.
+
+Likely causes:
+- The source IP appears to be probing sensitive web paths on the exposed application.
+
+Uncertainty notes:
+- Telemetry is limited to the bundled sample evidence and does not confirm operator intent.
+- The case summary is advisory only and requires human review before any incident classification.
+- Prompt-like text appeared in telemetry and was treated strictly as untrusted evidence.
+
+Suggested next steps:
+- Review the raw evidence and confirm whether the activity aligns with an approved administrative task.
+- Compare the web requests with reverse-proxy and WAF logs to determine whether the probing continued.
+- Document the analyst conclusion separately after human verification; do not treat this summary as a final verdict.
+
+## CASE-003
+
+- Severity: high
+- First seen: 2026-03-27T09:20:00Z
+- Last seen: 2026-03-27T09:20:20Z
+- Rule hits: encoded_powershell_execution, encoded_powershell_execution
+- ATT&CK: T1059.001
+
+Summary: CASE-003 contains 2 deterministic rule hits covering encoded_powershell_execution for principal lab_user; host wkstn-07 during 2026-03-27T09:20:00Z to 2026-03-27T09:20:20Z. The case warrants analyst review but does not imply a final incident decision.
+
+Likely causes:
+- Obfuscated PowerShell execution may reflect manual tradecraft or an unsafe script.
+
+Uncertainty notes:
+- Telemetry is limited to the bundled sample evidence and does not confirm operator intent.
+- The case summary is advisory only and requires human review before any incident classification.
+
+Suggested next steps:
+- Review the raw evidence and confirm whether the activity aligns with an approved administrative task.
+- Inspect the originating host timeline and validate whether the encoded PowerShell command matches known tooling.
+- Document the analyst conclusion separately after human verification; do not treat this summary as a final verdict.

demos/config-change-investigation-demo/README.md

@@ -58,6 +58,17 @@ Evidence is attached only when:
 - `artifacts/investigation_summary.json`
 - `artifacts/investigation_report.md`
 
+## Expected Run Summary
+
+The bundled sample run should report:
+
+- `4` normalized configuration changes
+- `3` risky changes
+- `3` investigations
+- `15` minute correlation window
+- `1` benign change that remains normalized but does not become an investigation
+- `1` risky break-glass investigation with no nearby supporting evidence
+
 ## Artifact Semantics
 
 - `change_events_normalized.json`: normalized config changes before any rule match is applied

demos/rule-evaluation-and-dedup-demo/README.md

@@ -59,6 +59,16 @@ That means repeated hits for the same rule can still be kept separately when the
 - `artifacts/dedup_explanations.json`
 - `artifacts/dedup_report.md`
 
+## Expected Run Summary
+
+The bundled sample run should report:
+
+- `10` raw rule hits
+- `6` retained alerts
+- `4` suppressed repeated hits
+- `4` rule/scope groups
+- `180` second cooldown
+
 ## Artifact Semantics
 
 - `rule_hits_before_dedup.json`: normalized raw hits with resolved cooldown scope and cooldown key

docs/design-notes.md

@@ -23,7 +23,7 @@ The implementation follows a narrow pipeline:
 - `pandas` is used for clarity and concise feature computation.
 - Rules stay threshold-based to emphasize detection semantics over model complexity.
 - Relative config paths are resolved from the repository root when the config lives under `configs/`.
-- Outputs are CSV and PNG because they are easy to inspect, diff, and embed in README material.
+- Outputs are CSV, JSON, and PNG: tables and summaries stay easy to diff, while PNG timelines are easy to inspect and embed in README material.
 - The notebook remains intentionally tiny so the packaged CLI pipeline stays the primary entrypoint.
 
 ## Non-goals