feat: Node I/O Inspector and Robust Large Payload Handling

[betterclever]

ShipSec Improvements: Node I/O Inspector & Workflow Overhaul

This branch (feature/node-io-inspector-and-fixes) introduces the Node I/O Inspector, a major enhancement to workflow observability, along with critical robustness fixes for handling large data payloads.

1. Node I/O Inspector & Registry

• Persistent Telemetry: Added a dedicated node_io database schema to store inputs and outputs for every node execution.
• Backend Service: Developed a new NodeIOService to manage the recording and retrieval of node data.
• Inspector API: Implemented REST endpoints to allow the frontend to visualize exactly what data passed through each node.
• Optimized Tracing: Refined the trace recording logic to provide real-time updates on node status (Started, Completed, Failed).

2. Robust Large Payload Handling (Data Spilling)

• Automatic Spilling: Implemented a "spill" mechanism for inputs and outputs exceeding 2MB. Large payloads are automatically moved to object storage (MinIO/S3), replacing the data in Temporal with a lightweight marker.
• Transparent Rehydration: Updated the activity runner to automatically download and "unspill" data before passing it to downstream components. Components receive the full data without needing to know it was spilled.
• UI Previews: Large spilled outputs now show truncated previews in the UI, with a "View Full Output" modal for deep inspection.

3. Docker Runner & Reliability Fixes

• Volume-Based Input Delivery: Switched from environment variables to file-based input delivery (input.json). This resolves E2BIG: argument list too long errors when passing massive datasets (e.g., thousands of Prowler findings) to components like the Script node.
• PTY Streaming Fixes:
  • Resolved a critical macOS permission issue for node-pty that prevented terminal log streaming.
  • Fixed stdin pollution in TTY mode to ensure clean logs.
• Robust Docker Detection: Added a proactively resolved Docker path utility to support multiple Docker backends (OrbStack, Homebrew, etc.).

4. Component Enhancements

• Prowler Scan: Integrated formal AWS credential contracts and optimized permissions for output volumes.
• Logic Script: Upgraded the script runner to handle massive JSON inputs via mounted volumes, enabling complex post-processing of large security scans.

5. Maintenance

• Standardized spill markers across the codebase.
• Updated core dependencies: node-pty, @ai-sdk, and @aws-sdk.

[chatgpt-codex-connector]

💡 Codex Review

studio/worker/src/temporal/input-resolver.ts

Lines 94 to 97 in ac99037

	if (portMetadata?.dataType) {
	const coercion = coerceValueForPort(portMetadata.dataType, resolved);
	if (coercion.ok) {
	params[targetKey] = coercion.value;

Skip coercion for spilled markers before unspill

When an upstream output is spilled, resolveInputValue returns a spill-marker object (with __spilled__/storageRef). The subsequent coerceValueForPort(...) runs before the activity unspills, so any non-JSON/any port (e.g., text, number, list) receives an object and fails coercion. That turns the input into a warning and ultimately a “missing required inputs” error, so large outputs (> spill threshold) can no longer flow to typed ports. Consider bypassing coercion when the resolved value is a spill marker (or deferring coercion until after unspill in the activity).

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".