decodingus-nexus is a critical, horizontally scalable component of the Decoding Us federated research platform. Implemented in Rust, its primary function is to act as the Aggregation Layer—the equivalent of a Big Graph Service (BGS) in federated architectures (e.g., Bluesky).
It ensures a complete, synchronized view of metadata by continuously polling and aggregating data from distributed PDS nodes within the custom Atmosphere environment. This aggregated data is then broadcast as a Firehose stream to the downstream App Layer components.
The project operates as a Sync Cluster, designed for high availability and horizontal scalability to efficiently handle the volume of metadata generated by the federated PDS nodes.
| Feature | Description |
|---|---|
| Language | Rust |
| Architecture | Horizontally Scalable Sync Cluster |
| Input Source | Metadata database from PDS nodes (in the custom Atmosphere) |
| Output Stream | Firehose data stream |
| Role | Central Aggregator (BGS Equivalent) |
🛣️ Deployment Phases Phase 1: Pilot & Direct Integration
The pilot phase will focus on validating synchronization and data integrity.
Implementation: A single decodingus-nexus instance will be deployed.
Data Flow: The Firehose data will be sent directly to the Decoding Us App Layer via API calls. This allows for rapid iteration and testing of the core synchronization logic.
Phase 2: Scale-Out & Production Readiness
The scale-out phase will introduce robust infrastructure components necessary for production-level federation and throughput.
Implementation: The full Sync Cluster architecture will be utilized for horizontal scalability.
Data Flow: The direct API calls will be replaced by a high-throughput Kafka middle layer, decoupling decodingus-nexus from the App Layer and providing durability, buffering, and increased fan-out capabilities for downstream consumers.
💻 Getting Started (WIP)
(Instructions for setting up the Rust environment, local database connections, and running the service will be added here.)