Skip to content

jfowler-cloud/forensics-ai

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

18 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

🔬 Forensics AI

Digital forensics pipeline for analyzing memory dumps, detecting threats with Sigma rules, and managing security incidents in real time

License: MIT Go C TypeScript

Built in 4 Hours Go Tests Frontend Tests Frontend Coverage Cloudscape Demo


Why This Project

After building Resume Tailor AI, Scaffold AI, Project Planner AI, and Career Path Architect, I wanted to explore a different domain entirely — one that required working closer to the metal.

Forensics AI is a deliberate departure from the Python/LangGraph/Bedrock stack used in the other projects. It demonstrates:

  • Systems programming in C — binary format parsing, raw memory layout, pointer arithmetic
  • Go for production services — concurrent ingestion, state machines, zero-dependency deployment
  • Full-stack integration — a C binary feeding a Go API feeding a React UI, all wired together
  • Domain depth — real forensic concepts: minidump streams, ELF segments, Sigma detection rules, MITRE ATT&CK tagging, incident lifecycle management
Resume Tailor AI Scaffold AI Project Planner AI Career Path Architect Forensics AI
Purpose Resume optimization AWS architecture design Project planning Career roadmaps Threat detection
Backend Python / Lambda Python / FastAPI Python / FastAPI Python / FastAPI Go + C + Python
Orchestration AWS Step Functions LangGraph LangGraph LangGraph Go state machine + LangGraph
AI Claude via Bedrock Claude via Bedrock Claude via Bedrock Claude via Bedrock Sigma rules + Claude via Bedrock
Frontend React 19 / Cloudscape Next.js 15 / Cloudscape Next.js 15 / Tailwind Next.js 15 / React Flow Next.js 15 / Cloudscape
Tests 212, 98% 126, ~67% 99, 86% 142, 99% Go 100% + 165 frontend

All five projects share production patterns — validation, rate limiting, error handling, comprehensive testing — but each uses a different architectural approach suited to its problem domain.


How It Works

memory-analyzer (C)  →  edr-pipeline (Go)  →  ai-analyst (Python)  →  web (Next.js)
   parse dump            detect & triage        LangGraph agents         visualize
  1. The C binary reads a real binary artifact from disk (Windows minidump, Linux ELF core, disk image)
  2. It parses the binary structure and extracts forensic findings — processes, modules, network connections, command history, deleted files
  3. It POSTs the structured JSON to the Go pipeline (or you use the interactive demo in the UI)
  4. The Go service runs Sigma detection rules against the findings, scores and classifies alerts, and auto-creates incidents for critical findings
  5. If an alert is critical AND scores ≥ 90, Go automatically escalates to the Python AI analyst — no human click required
  6. The Python LangGraph service runs four agents in sequence: triage → threat intel (MITRE ATT&CK) → assignment → report, with a human-in-the-loop pause before the final report
  7. The completed report is synced back to the Go incident as a formatted note
  8. The web UI visualizes everything in real time — alerts by severity, incident lifecycle, AI analyst reports rendered as formatted markdown

Screenshots

Landing Page & Dashboard

Landing Page Populated Dashboard
Landing Page Dashboard with Demo Data

Ingest Findings

Ingest Findings

Alerts & Incidents

Alerts Incidents
Alerts Incidents
Incident Ticket with AI Notes
Incident with AI Notes

Sigma Rules

Sigma Rules

AI Analyst

AI Analyst
AI Analyst
Auto Mode Results Human Review Mode Results
Auto Mode Human Review Results

Human-in-the-Loop Review

Pending Review Queue Review Form
Review Selection Review Form

Interactive Demo

Interactive Demo


Quick Start

git clone https://github.com/jfowler-cloud/forensics-ai.git
cd forensics-ai
./dev.sh

dev.sh starts all three services. The Go pipeline automatically escalates critical alerts (score ≥ 90) to the AI analyst via ANALYST_API_URL=http://localhost:8001. If the AI analyst isn't running, escalation is silently skipped and the demo falls back to posting directly.

Prerequisites

Run the demo script directly

cd apps/edr-pipeline
./demo/run_demo.sh

Auto-detects if the server is already running and skips build+start if so — safe to run against a live instance.

Analyze a real memory dump

cd apps/memory-analyzer
make build

# POST findings directly to the pipeline
./build/memory-analyzer --post http://localhost:8099/api/v1/findings /path/to/dump.dmp

# Write to JSON file instead
./build/memory-analyzer -t coredump -o results.json /path/to/core

# Enable file carving
./build/memory-analyzer --carve --carve-dir ./recovered/ /path/to/disk.img

Architecture

apps/memory-analyzer — C

The forensic parser. A compiled CLI binary that reads actual dump files from disk.

Why C: Memory forensics means reading raw binary formats — minidump stream directories, ELF program headers, MFT entries. C gives direct control over struct layout, pointer arithmetic, and byte-level parsing without a runtime in the way. The binary compiles to a single executable with no dependencies beyond libcurl (for HTTP POST) and cJSON (vendored). It runs on an air-gapped forensic workstation, inside Docker, or embedded in a larger toolchain.

Trade-offs vs Python: Python would be the obvious choice if this were doing ML-based anomaly detection — the ecosystem is unmatched. But for binary format parsing, Python's struct module and ctypes add friction that C eliminates. The cost is manual memory management: findings_free() must be called explicitly, and there's no garbage collector to catch leaks.

Trade-offs vs Rust: Rust would give memory safety without a GC — a genuine improvement over C for this use case. The trade-off is a steeper learning curve and a more complex build system. C was chosen here for portability and simplicity of the toolchain.

Supported formats:

Format Detection Validation Deep Parsing
Windows Minidump (.dmp) ✅ magic 0x504D444D 🚧 stream walking stubbed
Linux ELF Core ✅ magic 0x7F454C46 🚧 PT_NOTE parsing stubbed
Disk Image ✅ fallback 🚧 stubbed
SHA-256 🚧 stubbed (outputs zeros)
File Carving ✅ scaffolded 🚧 stubbed
HTTP POST (libcurl)
JSON output (cJSON)

Data model (contracts.h):

AnalysisResult
  ├── analysis_id, timestamp, source_type, source_path, sha256_hash
  └── Findings
        ├── processes[]       pid, ppid, name, path, cmdline, suspicious flag
        ├── modules[]         name, path, base_address, size, hash, injected flag
        ├── connections[]     pid, local/remote addr+port, protocol, state
        ├── commands[]        shell history with uid/timestamp
        ├── deleted_files[]   recoverable file metadata
        └── carved_files[]    files recovered from raw bytes

apps/edr-pipeline — Go

The detection and triage engine. A stateful HTTP service that receives findings, evaluates Sigma rules, and manages the incident lifecycle.

Why Go over Python: The pipeline handles concurrent ingestion, rule evaluation, and API requests simultaneously. Go's goroutines make that straightforward — the rate limiter, background eviction, and request handling all run concurrently with minimal boilerplate. The result is a single static binary (go build) with no runtime dependency. Compile-time type safety caught interface mismatches during development that would have been runtime errors in Python.

Python would win if the pipeline were doing ML inference — the ecosystem (numpy, transformers, LangChain) is unmatched and is exactly what the other projects in this portfolio use. Go's ML story is thin. For a high-throughput ingestion pipeline with no AI inference, Go is the better fit.

Why Go over Node: Node would make sense if the team wanted to share types between frontend and backend — one language across the stack is a real operational simplification. Go produces a smaller, faster binary with more predictable latency (no V8 JIT warm-up, no GC pauses at Node's scale). For a security tool where consistent response times matter, that's meaningful.

Key capabilities:

  • Sigma-style YAML detection rules evaluated against process names, cmdlines, network ports
  • Alert scoring (0–100) with severity classification (low / medium / high / critical)
  • MITRE ATT&CK tagging on every alert
  • Auto-incident creation when a critical alert fires
  • Automatic AI escalation — alerts with severity=critical AND score ≥ 90 are forwarded to the Python LangGraph analyst in a background goroutine (zero ingest latency impact). Controlled by ANALYST_API_URL env var; disabled if unset.
  • Incident state machine: new → triaging → investigating → containing → resolved → closed
  • Hot-reload rules from disk without restart
  • Audit logging per analysis ID
  • Rate limiting: 100 req/min general, 10 req/min for rule uploads
  • CORS middleware for browser-based UI access

Test coverage: 100% on all packages excluding the main.go entry point.

API:

GET  /health
POST /api/v1/findings          ingest analysis result
GET  /api/v1/findings/:id
GET  /api/v1/alerts            ?severity=critical&analysis_id=...
GET  /api/v1/alerts/:id
GET  /api/v1/incidents         ?status=triaging
POST /api/v1/incidents
PATCH /api/v1/incidents/:id    status transitions + assignee
POST /api/v1/incidents/:id/notes
GET  /api/v1/rules
POST /api/v1/rules             upload custom Sigma rule
POST /api/v1/rules/reload      hot-reload from disk
GET  /api/v1/audit/:analysisId

apps/ai-analyst — Python / FastAPI / LangGraph

The AI analyst service. Receives escalated findings from Go and runs a four-agent LangGraph workflow.

Why Python here: The AI inference layer is exactly where Python wins — LangChain, LangGraph, and the Bedrock SDK are first-class. The Go pipeline handles the high-throughput ingestion and rule evaluation; Python handles the LLM orchestration. Each language does what it's best at.

Agents (sequential):

Agent Role
triage_agent Classifies overall severity, extracts IOCs
threat_intel_agent Maps findings to MITRE ATT&CK techniques, profiles threat actor
assignment_agent Recommends analyst tier and priority (P1/P2/P3)
report_agent Generates incident title, executive summary, remediation steps, full markdown report
edr_sync_node POSTs the completed report back to the Go incident as a formatted note

Human-in-the-loop: After assignment_agent, the workflow pauses at a LangGraph interrupt. The /api/review endpoint resumes it with the human's decision (approve / modify / reject).

Deployment tiers (set DEPLOYMENT_TIER env var):

Tier Model
testing us.anthropic.claude-haiku-4-5-20251001-v1:0
optimized us.anthropic.claude-sonnet-4-5-20250929-v1:0
premium us.anthropic.claude-opus-4-5-20251101-v1:0

API:

GET  /health
POST /api/analyze                    start a new analysis run
POST /api/review                     resume after human review
GET  /api/status/{thread_id}         poll by thread ID
GET  /api/status/analysis/{id}       poll by EDR analysis ID (used by Go escalation)
GET  /api/workflows                  list all tracked workflows (?status=awaiting_review)

apps/web — Next.js 15 / React 19 / Cloudscape

The operations UI. Connects directly to the Go API from the browser.

Why Cloudscape: Consistent with Resume Tailor AI and Scaffold AI — AWS's open-source design system gives a dense, data-heavy UI out of the box. Tables, status indicators, badges, progress bars, and expandable sections are all built-in. It's designed for operational dashboards, which is exactly what this is. The trade-off is it's opinionated and visually AWS-flavored; not the right choice for a consumer product.

Pages:

Page Purpose
Dashboard Live metrics, top alerts by score, open incidents
Findings Manually ingest analysis results
Alerts Filterable alert table with severity and MITRE tags
Incidents Full lifecycle management — status transitions, notes, AI analyst reports
Sigma Rules View loaded rules, upload custom rules, hot-reload
AI Analyst Submit findings for AI analysis (auto or human-review mode)
Human Review Review and approve AI recommendations before report generation
Interactive Demo Step-by-step walkthrough of the full pipeline against the live API

Interactive Demo: 16 steps covering the full pipeline — health check → list rules → ingest routine findings (EDR-only) → ingest critical findings → list alerts → escalation threshold check → filter critical → auto-incident → triage → add note → upload custom rule → hot-reload → audit trail → AI triage (Go escalation or fallback) → human review → final health. "Run Full Demo" runs all steps sequentially with live API responses visible in each step card. Individual steps can be re-run independently.

The demo illustrates the escalation split: the routine workstation-02 payload fires medium/high alerts handled entirely by the EDR engine; the critical workstation-01 payload (mimikatz + C2 + injected DLL) crosses the threshold and is automatically handed to LangGraph.

Note: The analyst roster used in the demo (tier1-analyst@example.com, tier2-analyst@example.com, etc.) is fictional placeholder data. Replace ANALYST_ROSTER in apps/ai-analyst/src/analyst/graph/nodes.py with your own directory service in production.

Test coverage: 165 tests, >99% statement coverage.


Tech Stack

Layer Technology Why
Forensic parser C11, gcc, libcurl, cJSON Binary format parsing, zero-runtime deployment
Detection engine Go 1.22, chi router Concurrent ingestion, static binary, compile-time safety
AI escalation Go escalation package Fire-and-forget goroutine, zero ingest latency impact
AI analyst Python, FastAPI, LangGraph, LangChain-AWS LLM orchestration, human-in-the-loop, Bedrock
Web UI Next.js 15, React 19 App Router, RSC, fast iteration
Design system AWS Cloudscape Operational dashboard components out of the box
Testing (Go) stdlib testing, httptest No external test framework needed
Testing (frontend) Vitest, Testing Library, jsdom Fast, Vite-native, React 19 compatible
Build (C) Make, gcc Standard, portable, no build system overhead

Project Structure

forensics-ai/
├── apps/
│   ├── memory-analyzer/        C binary — parses dump files
│   │   ├── src/
│   │   │   ├── main.c
│   │   │   ├── analyzer.c      format detection, orchestration
│   │   │   ├── carver.c        file carving
│   │   │   ├── json_output.c   cJSON serialization
│   │   │   ├── http_client.c   libcurl POST
│   │   │   └── parsers/
│   │   │       ├── minidump.c
│   │   │       ├── coredump.c
│   │   │       └── diskimage.c
│   │   └── include/            headers / shared data contracts
│   │
│   ├── edr-pipeline/           Go service — detection & triage
│   │   ├── cmd/edr/            entry point (ANALYST_API_URL env var)
│   │   ├── internal/
│   │   │   ├── api/            HTTP handlers, middleware, router
│   │   │   ├── ingest/         findings receiver
│   │   │   ├── detection/      Sigma rule engine + scorer
│   │   │   ├── escalation/     AI escalation (critical + score≥90)
│   │   │   ├── incident/       state machine + store
│   │   │   └── audit/          audit logger
│   │   ├── rules/              YAML Sigma detection rules
│   │   └── demo/               end-to-end demo script + payloads
│   │
│   ├── ai-analyst/             Python service — LangGraph AI analyst
│   │   └── src/analyst/
│   │       ├── main.py         FastAPI app + thread registry
│   │       ├── config.py       deployment tier / model selection
│   │       └── graph/
│   │           ├── workflow.py LangGraph graph definition
│   │           ├── nodes.py    triage, threat_intel, assignment, report, edr_sync
│   │           └── state.py    AnalystState TypedDict
│   │
│   └── web/                    Next.js UI
│       ├── app/                pages: dashboard, findings, alerts,
│       │                              incidents, rules, analyst, demo
│       ├── components/         Navigation
│       └── lib/api.ts          typed API client
│
└── dev.sh                      starts all three services

Testing

# Go — all packages
cd apps/edr-pipeline
PATH=/snap/bin:$PATH go test ./internal/... -cover

# Go — with HTML coverage report
PATH=/snap/bin:$PATH go test ./internal/... -coverprofile=coverage.out
go tool cover -html=coverage.out

# Frontend
cd apps/web
npm test

# Frontend with coverage
npm run test:coverage

# C
cd apps/memory-analyzer
make test

# End-to-end demo (standalone)
cd apps/edr-pipeline
./demo/run_demo.sh

Related Projects

This project is part of a portfolio demonstrating different architectural approaches across different problem domains:


License

MIT — see LICENSE


Changelog

v1.0.0 — 2026-02-26

feat: AI analyst service, Go escalation, full Next.js frontend

✨ Features

  • apps/ai-analyst/ — Python / FastAPI / LangGraph AI analyst service

    • Four-agent sequential workflow: triage → threat intel (MITRE ATT&CK) → assignment → report
    • Human-in-the-loop interrupt after assignment; resume via /api/review
    • config.py — deployment tier pattern (testing / optimized / premium) with correct Claude 4.x model IDs
    • edr_sync_node — posts completed report back to Go incident as a formatted note
    • Thread registry (_analysis_threads) maps analysis_id → thread_id for Go escalation lookup
    • /api/status/analysis/{analysis_id} — lets Go-escalated runs be polled without knowing the thread ID
  • apps/edr-pipeline/internal/escalation/ — new Go package

    • ShouldEscalate()severity=critical AND score≥90
    • Escalator.Escalate() — fire-and-forget goroutine; zero ingest latency impact
    • New() returns nil when ANALYST_API_URL is unset (opt-in, no breaking change)
  • apps/edr-pipeline/internal/ingest/receiver.go — wired escalation into the critical-alert loop; NewReceiver accepts optional *escalation.Escalator (variadic, all existing tests unchanged)

  • apps/edr-pipeline/cmd/edr/main.go — reads ANALYST_API_URL, creates escalator, logs enabled/disabled on startup

  • dev.sh — starts all three services; passes ANALYST_API_URL=http://localhost:8001 to Go binary

  • apps/web/ — full Next.js 15 / React 19 / Cloudscape frontend

    • Dashboard, Findings, Alerts, Incidents, Sigma Rules, AI Analyst, Interactive Demo pages
    • app/demo/page.tsx — 16-step demo; two payloads (routine EDR-only vs critical AI-escalated); escalation threshold check step; ai-triage step polls Go-escalated thread first, falls back to direct POST
    • app/incidents/page.tsx — AI analyst notes rendered as formatted markdown (MarkdownNote inline renderer: headings, bold, code, lists, HR, paragraphs); regular notes unchanged
    • app/page.tsx — dashboard SpaceBetween key-prop fix (error Alert inside Container)

🧪 Tests

  • apps/edr-pipeline/internal/escalation/escalator_test.go — 6 tests: ShouldEscalate, nil config, default timeout, success, server error, unreachable
  • 165 frontend tests passing, >99% statement coverage

v0.4.0 — 2026-02-26 764d244

wip: demo folder, coverage explanation, README notes on paused state

✨ Features

  • demo/run_demo.sh — end-to-end shell script covering all 16 API paths; hits every major endpoint including state machine transitions, custom rule upload, hot-reload, and audit trail
  • demo/payload_workstation01.json — realistic mimikatz + C2 port 4444 + process injection findings payload
  • demo/custom_rule.json — PowerShell encoded command Sigma rule for upload demo step
  • apps/edr-pipeline/.gitignore — excludes build/ and coverage.out

📝 Docs

  • apps/edr-pipeline/README.md — WIP section with exact resume point and next steps; test coverage section explaining why 92% is the practical ceiling (entry point main.go excluded)

v0.3.0 — 2026-02-26 51cafc0

docs: add coverage badge and breakdown to README

📝 Docs

  • apps/edr-pipeline/README.md — coverage badge + per-package breakdown table added

v0.2.0 — 2026-02-26 8dc837c

test: comprehensive coverage — 92% total

🧪 Tests

  • internal/api/handlers_test.go — expanded from skeleton to full handler coverage (473 lines added)
  • internal/api/middleware_test.go — payload limit + rate limit middleware tests (97 lines)
  • internal/audit/logger_test.go — audit logger tests (54 lines)
  • internal/detection/engine_test.go — detection engine tests expanded (467 lines added)
  • internal/incident/workflow_test.go — state machine + store tests expanded (107 lines added)
  • internal/ingest/receiver_test.go — receiver tests expanded (291 lines added)

Coverage results: audit/incident 100%, ingest 98.5%, detection 98.4%, api 95.8%, total 92%


v0.1.0 — 2026-02-26 be72e69

fix: two rounds of critical review fixes

🐛 Round 1 fixes

  • handlers.goIngestFindings validation errors now return 400 with reason (was 500)
  • middleware.goRateLimitMiddleware strips TCP port from RemoteAddr before keying
  • detection/engine.goLoadRules atomically replaces rules slice (was appending on every reload call)
  • ingest/receiver.go — split single mutex into findMu + alertMu for independent maps
  • handlers.goTriggerAnalysis validates file_path is non-empty (was silently accepted)
  • handlers.goUpdateIncident invalid state transitions return 422, not 404
  • audit/logger.go — cap in-memory buffer at 10k entries with 10% eviction
  • router.go — added POST /api/v1/rules/reload for hot-reload without restart

🐛 Round 2 fixes

  • detection/engine.go — handle |contains, |startswith, |endswith Sigma field modifiers
  • detection/engine.go — multi-key selection is AND not OR (correct Sigma spec behaviour)
  • handlers.goAddRule rejects duplicate rule IDs with 400
  • ingest/receiver.go — cap findings store at 1000 entries with FIFO eviction
  • handlers.goIngestFindings response includes alert_count field
  • handlers.goListAlerts / ListIncidents return [] not null when empty
  • handlers.goHealthCheck reports rules_loaded, findings_stored, alerts_stored, uptime_seconds

📝 Docs

  • apps/edr-pipeline/README.md — full critical review documentation with both rounds

v0.0.1 — 2026-02-26 4f688b8

feat: implement edr-pipeline Go service

Initial implementation — 55 files, 3,193 lines added.

✨ edr-pipeline (Go)

  • internal/contracts/AnalysisResult, Findings, Alert, SigmaRule types
  • internal/audit/ — append-only JSON audit logger
  • internal/detection/ — Sigma rule engine (process name, suspicious flag, remote port matching) + threat scorer (0–100)
  • internal/incident/ — state machine workflow (new → triaging → investigating → containing → resolved → closed) + in-memory store
  • internal/ingest/Receiver (validate, store, evaluate, auto-incident on critical alerts) + input validator
  • internal/api/ — chi router, all handlers, payload limit + rate limit middleware
  • cmd/edr/ — graceful-shutdown HTTP server (EDR_PORT, EDR_RULES_DIR env vars)
  • rules/ — 3 Sigma rules: proc_injection, suspicious_network, credential_access
  • testdata/ — sample findings JSON fixture
  • Makefile, Dockerfile (multi-stage, alpine runtime)

✨ memory-analyzer (C)

  • include/contracts.h — shared data model: AnalysisResult, Findings, ProcessInfo, ModuleInfo, NetworkConnection, CommandHistoryEntry, DeletedFile, CarvedFile
  • src/analyzer.c — format auto-detection by magic bytes, orchestration
  • src/parsers/minidump.c — Windows minidump signature validation
  • src/parsers/coredump.c — ELF core dump magic validation
  • src/parsers/diskimage.c — disk image fallback parser
  • src/carver.c — file carving scaffolding
  • src/json_output.c — cJSON serialization matching Go contract
  • src/http_client.c — libcurl POST to EDR pipeline
  • tests/ — Unity test framework with test stubs for all modules
  • vendor/cJSON/ — vendored cJSON library
  • Makefile, Dockerfile

About

AI-powered forensics pipeline: EDR detection engine + memory analyzer

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors