Digital forensics pipeline for analyzing memory dumps, detecting threats with Sigma rules, and managing security incidents in real time
After building Resume Tailor AI, Scaffold AI, Project Planner AI, and Career Path Architect, I wanted to explore a different domain entirely — one that required working closer to the metal.
Forensics AI is a deliberate departure from the Python/LangGraph/Bedrock stack used in the other projects. It demonstrates:
- Systems programming in C — binary format parsing, raw memory layout, pointer arithmetic
- Go for production services — concurrent ingestion, state machines, zero-dependency deployment
- Full-stack integration — a C binary feeding a Go API feeding a React UI, all wired together
- Domain depth — real forensic concepts: minidump streams, ELF segments, Sigma detection rules, MITRE ATT&CK tagging, incident lifecycle management
| Resume Tailor AI | Scaffold AI | Project Planner AI | Career Path Architect | Forensics AI | |
|---|---|---|---|---|---|
| Purpose | Resume optimization | AWS architecture design | Project planning | Career roadmaps | Threat detection |
| Backend | Python / Lambda | Python / FastAPI | Python / FastAPI | Python / FastAPI | Go + C + Python |
| Orchestration | AWS Step Functions | LangGraph | LangGraph | LangGraph | Go state machine + LangGraph |
| AI | Claude via Bedrock | Claude via Bedrock | Claude via Bedrock | Claude via Bedrock | Sigma rules + Claude via Bedrock |
| Frontend | React 19 / Cloudscape | Next.js 15 / Cloudscape | Next.js 15 / Tailwind | Next.js 15 / React Flow | Next.js 15 / Cloudscape |
| Tests | 212, 98% | 126, ~67% | 99, 86% | 142, 99% | Go 100% + 165 frontend |
All five projects share production patterns — validation, rate limiting, error handling, comprehensive testing — but each uses a different architectural approach suited to its problem domain.
memory-analyzer (C) → edr-pipeline (Go) → ai-analyst (Python) → web (Next.js)
parse dump detect & triage LangGraph agents visualize
- The C binary reads a real binary artifact from disk (Windows minidump, Linux ELF core, disk image)
- It parses the binary structure and extracts forensic findings — processes, modules, network connections, command history, deleted files
- It POSTs the structured JSON to the Go pipeline (or you use the interactive demo in the UI)
- The Go service runs Sigma detection rules against the findings, scores and classifies alerts, and auto-creates incidents for critical findings
- If an alert is critical AND scores ≥ 90, Go automatically escalates to the Python AI analyst — no human click required
- The Python LangGraph service runs four agents in sequence: triage → threat intel (MITRE ATT&CK) → assignment → report, with a human-in-the-loop pause before the final report
- The completed report is synced back to the Go incident as a formatted note
- The web UI visualizes everything in real time — alerts by severity, incident lifecycle, AI analyst reports rendered as formatted markdown
Landing Page & Dashboard
| Landing Page | Populated Dashboard |
|---|---|
![]() |
![]() |
Ingest Findings
Alerts & Incidents
| Alerts | Incidents |
|---|---|
![]() |
![]() |
| Incident Ticket with AI Notes | |
|---|---|
![]() |
Sigma Rules
AI Analyst
| AI Analyst | |
|---|---|
![]() |
| Auto Mode Results | Human Review Mode Results |
|---|---|
![]() |
![]() |
Human-in-the-Loop Review
| Pending Review Queue | Review Form |
|---|---|
![]() |
![]() |
Interactive Demo
git clone https://github.com/jfowler-cloud/forensics-ai.git
cd forensics-ai
./dev.sh- EDR Pipeline API → http://localhost:8099
- AI Analyst API → http://localhost:8001 (requires
uv+ AWS credentials) - Web UI → http://localhost:3000
dev.sh starts all three services. The Go pipeline automatically escalates critical alerts (score ≥ 90) to the AI analyst via ANALYST_API_URL=http://localhost:8001. If the AI analyst isn't running, escalation is silently skipped and the demo falls back to posting directly.
- Go 1.22+ (
sudo snap install go --classicor https://golang.org/dl/) - Node.js 22+ and npm
- gcc + libcurl (for the C binary):
sudo apt install build-essential libcurl4-openssl-dev - Python 3.11+ with
uv(for the AI analyst): https://docs.astral.sh/uv/getting-started/installation/ - AWS credentials with Bedrock access (
aws sso loginor env vars)
cd apps/edr-pipeline
./demo/run_demo.shAuto-detects if the server is already running and skips build+start if so — safe to run against a live instance.
cd apps/memory-analyzer
make build
# POST findings directly to the pipeline
./build/memory-analyzer --post http://localhost:8099/api/v1/findings /path/to/dump.dmp
# Write to JSON file instead
./build/memory-analyzer -t coredump -o results.json /path/to/core
# Enable file carving
./build/memory-analyzer --carve --carve-dir ./recovered/ /path/to/disk.imgThe forensic parser. A compiled CLI binary that reads actual dump files from disk.
Why C: Memory forensics means reading raw binary formats — minidump stream directories, ELF program headers, MFT entries. C gives direct control over struct layout, pointer arithmetic, and byte-level parsing without a runtime in the way. The binary compiles to a single executable with no dependencies beyond libcurl (for HTTP POST) and cJSON (vendored). It runs on an air-gapped forensic workstation, inside Docker, or embedded in a larger toolchain.
Trade-offs vs Python:
Python would be the obvious choice if this were doing ML-based anomaly detection — the ecosystem is unmatched. But for binary format parsing, Python's struct module and ctypes add friction that C eliminates. The cost is manual memory management: findings_free() must be called explicitly, and there's no garbage collector to catch leaks.
Trade-offs vs Rust: Rust would give memory safety without a GC — a genuine improvement over C for this use case. The trade-off is a steeper learning curve and a more complex build system. C was chosen here for portability and simplicity of the toolchain.
Supported formats:
| Format | Detection | Validation | Deep Parsing |
|---|---|---|---|
Windows Minidump (.dmp) |
✅ magic 0x504D444D |
✅ | 🚧 stream walking stubbed |
| Linux ELF Core | ✅ magic 0x7F454C46 |
✅ | 🚧 PT_NOTE parsing stubbed |
| Disk Image | ✅ fallback | — | 🚧 stubbed |
| SHA-256 | — | — | 🚧 stubbed (outputs zeros) |
| File Carving | ✅ scaffolded | — | 🚧 stubbed |
| HTTP POST (libcurl) | ✅ | ✅ | ✅ |
| JSON output (cJSON) | ✅ | ✅ | ✅ |
Data model (contracts.h):
AnalysisResult
├── analysis_id, timestamp, source_type, source_path, sha256_hash
└── Findings
├── processes[] pid, ppid, name, path, cmdline, suspicious flag
├── modules[] name, path, base_address, size, hash, injected flag
├── connections[] pid, local/remote addr+port, protocol, state
├── commands[] shell history with uid/timestamp
├── deleted_files[] recoverable file metadata
└── carved_files[] files recovered from raw bytes
The detection and triage engine. A stateful HTTP service that receives findings, evaluates Sigma rules, and manages the incident lifecycle.
Why Go over Python:
The pipeline handles concurrent ingestion, rule evaluation, and API requests simultaneously. Go's goroutines make that straightforward — the rate limiter, background eviction, and request handling all run concurrently with minimal boilerplate. The result is a single static binary (go build) with no runtime dependency. Compile-time type safety caught interface mismatches during development that would have been runtime errors in Python.
Python would win if the pipeline were doing ML inference — the ecosystem (numpy, transformers, LangChain) is unmatched and is exactly what the other projects in this portfolio use. Go's ML story is thin. For a high-throughput ingestion pipeline with no AI inference, Go is the better fit.
Why Go over Node: Node would make sense if the team wanted to share types between frontend and backend — one language across the stack is a real operational simplification. Go produces a smaller, faster binary with more predictable latency (no V8 JIT warm-up, no GC pauses at Node's scale). For a security tool where consistent response times matter, that's meaningful.
Key capabilities:
- Sigma-style YAML detection rules evaluated against process names, cmdlines, network ports
- Alert scoring (0–100) with severity classification (low / medium / high / critical)
- MITRE ATT&CK tagging on every alert
- Auto-incident creation when a critical alert fires
- Automatic AI escalation — alerts with
severity=criticalANDscore ≥ 90are forwarded to the Python LangGraph analyst in a background goroutine (zero ingest latency impact). Controlled byANALYST_API_URLenv var; disabled if unset. - Incident state machine:
new → triaging → investigating → containing → resolved → closed - Hot-reload rules from disk without restart
- Audit logging per analysis ID
- Rate limiting: 100 req/min general, 10 req/min for rule uploads
- CORS middleware for browser-based UI access
Test coverage: 100% on all packages excluding the main.go entry point.
API:
GET /health
POST /api/v1/findings ingest analysis result
GET /api/v1/findings/:id
GET /api/v1/alerts ?severity=critical&analysis_id=...
GET /api/v1/alerts/:id
GET /api/v1/incidents ?status=triaging
POST /api/v1/incidents
PATCH /api/v1/incidents/:id status transitions + assignee
POST /api/v1/incidents/:id/notes
GET /api/v1/rules
POST /api/v1/rules upload custom Sigma rule
POST /api/v1/rules/reload hot-reload from disk
GET /api/v1/audit/:analysisId
The AI analyst service. Receives escalated findings from Go and runs a four-agent LangGraph workflow.
Why Python here: The AI inference layer is exactly where Python wins — LangChain, LangGraph, and the Bedrock SDK are first-class. The Go pipeline handles the high-throughput ingestion and rule evaluation; Python handles the LLM orchestration. Each language does what it's best at.
Agents (sequential):
| Agent | Role |
|---|---|
triage_agent |
Classifies overall severity, extracts IOCs |
threat_intel_agent |
Maps findings to MITRE ATT&CK techniques, profiles threat actor |
assignment_agent |
Recommends analyst tier and priority (P1/P2/P3) |
report_agent |
Generates incident title, executive summary, remediation steps, full markdown report |
edr_sync_node |
POSTs the completed report back to the Go incident as a formatted note |
Human-in-the-loop: After assignment_agent, the workflow pauses at a LangGraph interrupt. The /api/review endpoint resumes it with the human's decision (approve / modify / reject).
Deployment tiers (set DEPLOYMENT_TIER env var):
| Tier | Model |
|---|---|
testing |
us.anthropic.claude-haiku-4-5-20251001-v1:0 |
optimized |
us.anthropic.claude-sonnet-4-5-20250929-v1:0 |
premium |
us.anthropic.claude-opus-4-5-20251101-v1:0 |
API:
GET /health
POST /api/analyze start a new analysis run
POST /api/review resume after human review
GET /api/status/{thread_id} poll by thread ID
GET /api/status/analysis/{id} poll by EDR analysis ID (used by Go escalation)
GET /api/workflows list all tracked workflows (?status=awaiting_review)
The operations UI. Connects directly to the Go API from the browser.
Why Cloudscape: Consistent with Resume Tailor AI and Scaffold AI — AWS's open-source design system gives a dense, data-heavy UI out of the box. Tables, status indicators, badges, progress bars, and expandable sections are all built-in. It's designed for operational dashboards, which is exactly what this is. The trade-off is it's opinionated and visually AWS-flavored; not the right choice for a consumer product.
Pages:
| Page | Purpose |
|---|---|
| Dashboard | Live metrics, top alerts by score, open incidents |
| Findings | Manually ingest analysis results |
| Alerts | Filterable alert table with severity and MITRE tags |
| Incidents | Full lifecycle management — status transitions, notes, AI analyst reports |
| Sigma Rules | View loaded rules, upload custom rules, hot-reload |
| AI Analyst | Submit findings for AI analysis (auto or human-review mode) |
| Human Review | Review and approve AI recommendations before report generation |
| Interactive Demo | Step-by-step walkthrough of the full pipeline against the live API |
Interactive Demo: 16 steps covering the full pipeline — health check → list rules → ingest routine findings (EDR-only) → ingest critical findings → list alerts → escalation threshold check → filter critical → auto-incident → triage → add note → upload custom rule → hot-reload → audit trail → AI triage (Go escalation or fallback) → human review → final health. "Run Full Demo" runs all steps sequentially with live API responses visible in each step card. Individual steps can be re-run independently.
The demo illustrates the escalation split: the routine workstation-02 payload fires medium/high alerts handled entirely by the EDR engine; the critical workstation-01 payload (mimikatz + C2 + injected DLL) crosses the threshold and is automatically handed to LangGraph.
Note: The analyst roster used in the demo (
tier1-analyst@example.com,tier2-analyst@example.com, etc.) is fictional placeholder data. ReplaceANALYST_ROSTERinapps/ai-analyst/src/analyst/graph/nodes.pywith your own directory service in production.
Test coverage: 165 tests, >99% statement coverage.
| Layer | Technology | Why |
|---|---|---|
| Forensic parser | C11, gcc, libcurl, cJSON | Binary format parsing, zero-runtime deployment |
| Detection engine | Go 1.22, chi router | Concurrent ingestion, static binary, compile-time safety |
| AI escalation | Go escalation package |
Fire-and-forget goroutine, zero ingest latency impact |
| AI analyst | Python, FastAPI, LangGraph, LangChain-AWS | LLM orchestration, human-in-the-loop, Bedrock |
| Web UI | Next.js 15, React 19 | App Router, RSC, fast iteration |
| Design system | AWS Cloudscape | Operational dashboard components out of the box |
| Testing (Go) | stdlib testing, httptest |
No external test framework needed |
| Testing (frontend) | Vitest, Testing Library, jsdom | Fast, Vite-native, React 19 compatible |
| Build (C) | Make, gcc | Standard, portable, no build system overhead |
forensics-ai/
├── apps/
│ ├── memory-analyzer/ C binary — parses dump files
│ │ ├── src/
│ │ │ ├── main.c
│ │ │ ├── analyzer.c format detection, orchestration
│ │ │ ├── carver.c file carving
│ │ │ ├── json_output.c cJSON serialization
│ │ │ ├── http_client.c libcurl POST
│ │ │ └── parsers/
│ │ │ ├── minidump.c
│ │ │ ├── coredump.c
│ │ │ └── diskimage.c
│ │ └── include/ headers / shared data contracts
│ │
│ ├── edr-pipeline/ Go service — detection & triage
│ │ ├── cmd/edr/ entry point (ANALYST_API_URL env var)
│ │ ├── internal/
│ │ │ ├── api/ HTTP handlers, middleware, router
│ │ │ ├── ingest/ findings receiver
│ │ │ ├── detection/ Sigma rule engine + scorer
│ │ │ ├── escalation/ AI escalation (critical + score≥90)
│ │ │ ├── incident/ state machine + store
│ │ │ └── audit/ audit logger
│ │ ├── rules/ YAML Sigma detection rules
│ │ └── demo/ end-to-end demo script + payloads
│ │
│ ├── ai-analyst/ Python service — LangGraph AI analyst
│ │ └── src/analyst/
│ │ ├── main.py FastAPI app + thread registry
│ │ ├── config.py deployment tier / model selection
│ │ └── graph/
│ │ ├── workflow.py LangGraph graph definition
│ │ ├── nodes.py triage, threat_intel, assignment, report, edr_sync
│ │ └── state.py AnalystState TypedDict
│ │
│ └── web/ Next.js UI
│ ├── app/ pages: dashboard, findings, alerts,
│ │ incidents, rules, analyst, demo
│ ├── components/ Navigation
│ └── lib/api.ts typed API client
│
└── dev.sh starts all three services
# Go — all packages
cd apps/edr-pipeline
PATH=/snap/bin:$PATH go test ./internal/... -cover
# Go — with HTML coverage report
PATH=/snap/bin:$PATH go test ./internal/... -coverprofile=coverage.out
go tool cover -html=coverage.out
# Frontend
cd apps/web
npm test
# Frontend with coverage
npm run test:coverage
# C
cd apps/memory-analyzer
make test
# End-to-end demo (standalone)
cd apps/edr-pipeline
./demo/run_demo.shThis project is part of a portfolio demonstrating different architectural approaches across different problem domains:
- Resume Tailor AI — AWS serverless, Step Functions, Claude Opus 4.5
- Scaffold AI — LangGraph multi-agent, AWS architecture generation
- Project Planner AI — LangGraph, streaming SSE, Scaffold AI integration
- Career Path Architect — 6-agent LangGraph system, visual roadmaps
MIT — see LICENSE
feat: AI analyst service, Go escalation, full Next.js frontend
✨ Features
-
apps/ai-analyst/— Python / FastAPI / LangGraph AI analyst service- Four-agent sequential workflow: triage → threat intel (MITRE ATT&CK) → assignment → report
- Human-in-the-loop interrupt after assignment; resume via
/api/review config.py— deployment tier pattern (testing / optimized / premium) with correct Claude 4.x model IDsedr_sync_node— posts completed report back to Go incident as a formatted note- Thread registry (
_analysis_threads) mapsanalysis_id → thread_idfor Go escalation lookup /api/status/analysis/{analysis_id}— lets Go-escalated runs be polled without knowing the thread ID
-
apps/edr-pipeline/internal/escalation/— new Go packageShouldEscalate()—severity=critical AND score≥90Escalator.Escalate()— fire-and-forget goroutine; zero ingest latency impactNew()returns nil whenANALYST_API_URLis unset (opt-in, no breaking change)
-
apps/edr-pipeline/internal/ingest/receiver.go— wired escalation into the critical-alert loop;NewReceiveraccepts optional*escalation.Escalator(variadic, all existing tests unchanged) -
apps/edr-pipeline/cmd/edr/main.go— readsANALYST_API_URL, creates escalator, logs enabled/disabled on startup -
dev.sh— starts all three services; passesANALYST_API_URL=http://localhost:8001to Go binary -
apps/web/— full Next.js 15 / React 19 / Cloudscape frontend- Dashboard, Findings, Alerts, Incidents, Sigma Rules, AI Analyst, Interactive Demo pages
app/demo/page.tsx— 16-step demo; two payloads (routine EDR-only vs critical AI-escalated); escalation threshold check step;ai-triagestep polls Go-escalated thread first, falls back to direct POSTapp/incidents/page.tsx— AI analyst notes rendered as formatted markdown (MarkdownNoteinline renderer: headings, bold, code, lists, HR, paragraphs); regular notes unchangedapp/page.tsx— dashboard SpaceBetween key-prop fix (error Alert inside Container)
🧪 Tests
apps/edr-pipeline/internal/escalation/escalator_test.go— 6 tests:ShouldEscalate, nil config, default timeout, success, server error, unreachable- 165 frontend tests passing, >99% statement coverage
wip: demo folder, coverage explanation, README notes on paused state
✨ Features
demo/run_demo.sh— end-to-end shell script covering all 16 API paths; hits every major endpoint including state machine transitions, custom rule upload, hot-reload, and audit traildemo/payload_workstation01.json— realistic mimikatz + C2 port 4444 + process injection findings payloaddemo/custom_rule.json— PowerShell encoded command Sigma rule for upload demo stepapps/edr-pipeline/.gitignore— excludesbuild/andcoverage.out
📝 Docs
apps/edr-pipeline/README.md— WIP section with exact resume point and next steps; test coverage section explaining why 92% is the practical ceiling (entry pointmain.goexcluded)
docs: add coverage badge and breakdown to README
📝 Docs
apps/edr-pipeline/README.md— coverage badge + per-package breakdown table added
test: comprehensive coverage — 92% total
🧪 Tests
internal/api/handlers_test.go— expanded from skeleton to full handler coverage (473 lines added)internal/api/middleware_test.go— payload limit + rate limit middleware tests (97 lines)internal/audit/logger_test.go— audit logger tests (54 lines)internal/detection/engine_test.go— detection engine tests expanded (467 lines added)internal/incident/workflow_test.go— state machine + store tests expanded (107 lines added)internal/ingest/receiver_test.go— receiver tests expanded (291 lines added)
Coverage results: audit/incident 100%, ingest 98.5%, detection 98.4%, api 95.8%, total 92%
fix: two rounds of critical review fixes
🐛 Round 1 fixes
handlers.go—IngestFindingsvalidation errors now return 400 with reason (was 500)middleware.go—RateLimitMiddlewarestrips TCP port fromRemoteAddrbefore keyingdetection/engine.go—LoadRulesatomically replaces rules slice (was appending on every reload call)ingest/receiver.go— split single mutex intofindMu+alertMufor independent mapshandlers.go—TriggerAnalysisvalidatesfile_pathis non-empty (was silently accepted)handlers.go—UpdateIncidentinvalid state transitions return 422, not 404audit/logger.go— cap in-memory buffer at 10k entries with 10% evictionrouter.go— addedPOST /api/v1/rules/reloadfor hot-reload without restart
🐛 Round 2 fixes
detection/engine.go— handle|contains,|startswith,|endswithSigma field modifiersdetection/engine.go— multi-key selection is AND not OR (correct Sigma spec behaviour)handlers.go—AddRulerejects duplicate rule IDs with 400ingest/receiver.go— cap findings store at 1000 entries with FIFO evictionhandlers.go—IngestFindingsresponse includesalert_countfieldhandlers.go—ListAlerts/ListIncidentsreturn[]notnullwhen emptyhandlers.go—HealthCheckreportsrules_loaded,findings_stored,alerts_stored,uptime_seconds
📝 Docs
apps/edr-pipeline/README.md— full critical review documentation with both rounds
feat: implement edr-pipeline Go service
Initial implementation — 55 files, 3,193 lines added.
✨ edr-pipeline (Go)
internal/contracts/—AnalysisResult,Findings,Alert,SigmaRuletypesinternal/audit/— append-only JSON audit loggerinternal/detection/— Sigma rule engine (process name, suspicious flag, remote port matching) + threat scorer (0–100)internal/incident/— state machine workflow (new → triaging → investigating → containing → resolved → closed) + in-memory storeinternal/ingest/—Receiver(validate, store, evaluate, auto-incident on critical alerts) + input validatorinternal/api/— chi router, all handlers, payload limit + rate limit middlewarecmd/edr/— graceful-shutdown HTTP server (EDR_PORT,EDR_RULES_DIRenv vars)rules/— 3 Sigma rules:proc_injection,suspicious_network,credential_accesstestdata/— sample findings JSON fixtureMakefile,Dockerfile(multi-stage, alpine runtime)
✨ memory-analyzer (C)
include/contracts.h— shared data model:AnalysisResult,Findings,ProcessInfo,ModuleInfo,NetworkConnection,CommandHistoryEntry,DeletedFile,CarvedFilesrc/analyzer.c— format auto-detection by magic bytes, orchestrationsrc/parsers/minidump.c— Windows minidump signature validationsrc/parsers/coredump.c— ELF core dump magic validationsrc/parsers/diskimage.c— disk image fallback parsersrc/carver.c— file carving scaffoldingsrc/json_output.c— cJSON serialization matching Go contractsrc/http_client.c— libcurl POST to EDR pipelinetests/— Unity test framework with test stubs for all modulesvendor/cJSON/— vendored cJSON libraryMakefile,Dockerfile












