[Feature] Implement Sidecar and Discovery Proxy for First Occurrence Data Collection (FODC)

[hanahmily]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

Feature Request

This issue tracks the implementation of two core components for the BanyanDB First Occurrence Data Collection (FODC) system: the FODC Sidecar and the Discovery Proxy. These components are essential for building an automated, state-aware forensics and diagnostics architecture.

Background

The BanyanDB FODC system is designed to capture the entire context of a crash or stall the moment it happens, enabling AI agents to resolve complex runtime issues without human reproduction. The architecture relies on a sidecar pattern for data collection and a discovery proxy for cluster-wide topology management and data access.

1. FODC Sidecar ("The Black Box")

The FODC Sidecar runs alongside each BanyanDB instance and is responsible for monitoring, recording, and coordinating diagnostic data collection.

Responsibilities:

• Watchdog: Polls metrics from the BanyanDB container and detects "death rattles" (e.g., signals, file-based triggers) that indicate an impending or current failure.
• Flight Recorder: Buffers metrics data in a circular buffer within its own memory space. This ensures data survives if the main BanyanDB process crashes.

2. Discovery Proxy

The Discovery Proxy is a standalone service that provides a centralized view of the BanyanDB cluster and serves as a gateway to diagnostic data.

Responsibilities:

• Topology Discovery and Recording: Continuously discovers and records the BanyanDB deployment topology. This includes node locations, shard assignments, and replication configurations. The topology should be versioned.
• Watchdog Data Access Gateway: Provides a unified API for accessing diagnostic data from all sidecars in the cluster. This abstracts the complexity of communicating with individual sidecars.

Implementation Tasks

• Implement the sidecar container with watchdog functionality.
• Implement the proxy service.
• Conduct e2e tests to ensure the entire data collection and access pipeline works as expected.
• Update Helm chart to support sidecar and proxy deployment

Use case

No response

Related issues

No response

Are you willing to submit a pull request to implement this on your own?

• Yes I am willing to submit a pull request on my own!

Code of Conduct

• I agree to follow this project's Code of Conduct

[wu-sheng]

FODC Proxy should start from leveraging the K8s API to watch the BanyanDB relative deployment, and collect info from sidecars of all deployments/stateful nodes. Each pod should be expected to have one FODC sidecar.
@Fine0830 Please share the exposed configuration structure here, to make sure we are in the same page about the overview of the whole deployment structure.

[wu-sheng]

FODC has been added.