|
14 | 14 |
|
15 | 15 | --- |
16 | 16 |
|
17 | | -# Exosphere: Distributed AI Workflow Infrastructure |
| 17 | +# Exosphere: Reliability Runtime for AI Agents |
18 | 18 |
|
19 | | -**Exosphere** is an open-source, Kubernetes-native infrastructure platform designed to run distributed AI workflows and autonomous agents at scale. Built with Python and based on a flexible node-based architecture, Exosphere enables developers to create, deploy, and manage robust AI workflows that can handle large-scale data processing and long-running operations. |
| 19 | +**Exosphere** is a lightweight runtime to make AI agents resilient to failure and infinite scaling across distributed compute. With a few changes to your existing agent code, take your agent from demo to deployment. |
20 | 20 |
|
21 | | - |
22 | | - |
23 | | -## 🚀 What Exosphere Can Do |
| 21 | +## Why Exosphere? |
24 | 22 |
|
25 | 23 | Exosphere provides a powerful foundation for building and orchestrating AI applications with these key capabilities: |
26 | 24 |
|
27 | | -### **Reliable AI Workflows at Scale** |
28 | | -- **Infinite Parallel Agents**: Run multiple AI agents simultaneously across distributed infrastructure |
29 | | -- **Dynamic State Management**: Create and manage state at runtime with persistent storage |
30 | | -- **Fault Tolerance**: Built-in failure handling and recovery mechanisms for production reliability |
31 | | -- **Core Concepts**: Fanout, Unite, Signals, Retry Policy, Store, Triggers |
| 25 | +| Feature | Description | |
| 26 | +|---------|-------------| |
| 27 | +| [**Lightweight Runtime**](https://docs.exosphere.host/exosphere/architecture) | Execute workflows reliably with minimal overhead across distributed infrastructure using a state-based execution model. | |
| 28 | +| [**Inbuilt Failure Handling**](https://docs.exosphere.host/exosphere/retry-policy) | Built-in retry policies with exponential backoff and jitter strategies for resilient, production-grade execution. | |
| 29 | +| [**Infinite Parallel Agents**](https://docs.exosphere.host/exosphere/fanout) | Scale to unlimited parallel agents with automatic load distribution and dynamic fanout at runtime. | |
| 30 | +| [**Dynamic Execution Graphs**](https://docs.exosphere.host/exosphere/concepts) | Durable execution designed for agentic flows with node-based control of execution. | |
| 31 | +| [**Native State Persistence**](https://docs.exosphere.host/exosphere/store) | Persist workflow state across restarts and failures with graph-level key-value storage. | |
| 32 | +| [**Observability**](https://docs.exosphere.host/exosphere/dashboard) | Visual monitoring, debugging, and management of workflows with real-time execution tracking. | |
32 | 33 |
|
33 | | -### **Smooth Developer Experience** |
34 | | -- **Plug-and-Play Nodes**: Create reusable, atomic workflow components that can be mixed and matched |
35 | | -- **Python-First**: Native Python support with Pydantic models for type-safe inputs/outputs |
36 | | -- **Interactive Dashboard**: Visual workflow management, monitoring, and debugging tools |
| 34 | +Whether you're building data pipelines, AI agents, or complex workflow orchestrations, Exosphere provides the infrastructure backbone to make your AI applications production-ready and scalable. |
37 | 35 |
|
38 | | -### **Production Ready Infrastructure** |
39 | | -- **Kubernetes Native**: Deploy seamlessly on Kubernetes clusters for enterprise-grade scalability |
40 | | -- **State Persistence**: Maintain workflow state across restarts and failures |
41 | | -- **API Integration**: Connect to external services and APIs through configurable nodes |
42 | 36 |
|
43 | | -### **Built for AI Agents** |
44 | | -- **Autonomous Execution**: Build agents that can make decisions and execute complex workflows |
45 | | -- **Data Processing**: Handle large datasets with distributed processing capabilities |
46 | | -- **Long-Running Operations**: Support for workflows that run for hours, days, or indefinitely |
| 37 | +--- |
47 | 38 |
|
48 | | -Whether you're building data pipelines, AI agents, or complex workflow orchestrations, Exosphere provides the infrastructure backbone to make your AI applications production-ready and scalable. |
| 39 | +## Run Your First Agent |
49 | 40 |
|
50 | | -## 🎯 Use Cases & Applications |
| 41 | +| Step | Description | |
| 42 | +|------|-------------| |
| 43 | +| [**Getting Started**](https://docs.exosphere.host/getting-started) | Get the Exosphere State Manager and Dashboard running locally for development. | |
| 44 | +| [**Run Your First Node**](https://docs.exosphere.host/exosphere/register-node) | Create your first node and register it with the Exosphere runtime. | |
| 45 | +| [**Trigger Agent**](https://docs.exosphere.host/exosphere/trigger-graph) | Learn how to trigger your agent workflows and manage execution flows. | |
| 46 | +| [**Deploy and Monitor**](https://docs.exosphere.host/exosphere/dashboard) | Deploy your agents and monitor their execution with the visual dashboard. | |
51 | 47 |
|
52 | | -Exosphere is perfect for a wide range of AI and automation scenarios: |
| 48 | + |
53 | 49 |
|
54 | | -- **Data Processing & ETL Pipelines** |
55 | | -- **AI Agent Orchestration** |
56 | | -- **Content Generation & Analysis** |
57 | | -- **API Integration & Automation** |
| 50 | +--- |
58 | 51 |
|
59 | | -## Architecture Overview |
| 52 | +## Architecture Overview |
60 | 53 |
|
61 | 54 | Exosphere is built on a flexible, node-based architecture that makes it easy to create complex workflows: |
62 | 55 |
|
63 | 56 |  |
64 | 57 |
|
65 | | -### **Core Components** |
| 58 | +### Core Components |
66 | 59 |
|
67 | | -- **Nodes**: Atomic, reusable units of work that can be AI agents, API calls, data processors, or any custom logic |
68 | | -- **Runtime**: The execution environment that manages and orchestrates your nodes |
69 | | -- **State Manager**: Handles persistent state across workflow executions |
70 | | -- **Dashboard**: Visual interface for monitoring and managing workflows |
71 | | -- **Graphs**: Define the flow and dependencies between nodes |
| 60 | +| Component | Description | |
| 61 | +|-----------|-------------| |
| 62 | +| **Nodes** | Atomic, reusable units of work that can be AI agents, API calls, data processors, or any custom logic | |
| 63 | +| **Runtime** | The execution environment that manages and orchestrates your nodes | |
| 64 | +| **State Manager** | Handles persistent state across workflow executions | |
| 65 | +| **Dashboard** | Visual interface for monitoring and managing workflows | |
| 66 | +| **Graphs** | Define the flow and dependencies between nodes | |
72 | 67 |
|
73 | | -### **Key Concepts** |
| 68 | +### Key Concepts |
74 | 69 |
|
75 | 70 |  |
76 | 71 |
|
77 | | -- **Fanout**: Distribute work across multiple parallel instances of a node |
78 | | -- **Unite**: Combine results from multiple parallel executions |
79 | | -- **Signals**: Inter-node communication and event handling |
80 | | -- **Retry Policy**: Configurable failure handling and recovery |
81 | | -- **Store**: Persistent storage for workflow state and data |
82 | | -- **Triggers**: Automatic scheduling with cron expressions |
83 | | - |
84 | | -## ⏰ Automatic Scheduling Example |
85 | | - |
86 | | -Schedule your workflows to run automatically using cron expressions: |
87 | | - |
88 | | -!!! info "Beta Feature" |
89 | | - Available in `beta-latest` Docker tag and SDK version `0.0.3b1` |
90 | | - |
91 | | -```python |
92 | | -from exospherehost import StateManager, GraphNodeModel, CronTrigger |
93 | | - |
94 | | -# Define triggers for automatic execution |
95 | | -triggers = [ |
96 | | - CronTrigger(expression="0 9 * * 1-5"), # Every weekday at 9 AM |
97 | | - CronTrigger(expression="0 */6 * * *") # Every 6 hours |
98 | | -] |
99 | | - |
100 | | -# Create graph with automatic scheduling |
101 | | -result = await state_manager.upsert_graph( |
102 | | - graph_name="data-pipeline", |
103 | | - graph_nodes=graph_nodes, |
104 | | - secrets={"api_key": "your-key"}, |
105 | | - triggers=triggers # Enable automatic execution (Beta) |
106 | | -) |
107 | | -``` |
108 | | - |
109 | | -### **Deployment Options** |
110 | | - |
111 | | -- **Local Development**: Run with Docker Compose for quick setup |
112 | | -- **Kubernetes**: Production-ready deployment on K8s clusters |
113 | | -- **Cloud**: Deploy on any cloud provider with Kubernetes support |
114 | | - |
115 | | -## 🚀 Getting Started |
116 | | - |
117 | | -### Prerequisites |
118 | | - |
119 | | -- Python 3.12+ |
120 | | -- [uv](https://docs.astral.sh/uv/) (recommended) or pip |
121 | | - |
122 | | -### Step 1: Installation |
123 | | - |
124 | | -```bash |
125 | | -uv add exospherehost |
126 | | -``` |
127 | | - |
128 | | -### Step 2: Create Your First Node |
129 | | - |
130 | | - Each node is an atomic, reusable unit in Exosphere. Once registered, you can plug it into any workflow. Nodes can be AI agents, API calls, data processors, or any custom logic you want to execute. |
131 | | - |
132 | | - ```python |
133 | | - from exospherehost import BaseNode |
134 | | - from pydantic import BaseModel |
135 | | - |
136 | | - class CityAnalyzerNode(BaseNode): |
137 | | - """A node that analyzes and describes a city using AI""" |
138 | | - |
139 | | - class Inputs(BaseModel): |
140 | | - city: str |
141 | | - analysis_type: str = "general" # general, tourism, business, etc. |
142 | | - |
143 | | - class Outputs(BaseModel): |
144 | | - description: str |
145 | | - key_features: str |
146 | | - score: str |
147 | | - |
148 | | - class Secrets(BaseModel): |
149 | | - openai_api_key: str # Optional: for AI-powered analysis |
150 | | - |
151 | | - async def execute(self) -> Outputs: |
152 | | - # Your custom logic here - could be: |
153 | | - # - AI agent calls |
154 | | - # - API requests |
155 | | - # - Data processing |
156 | | - # - Database queries |
157 | | - # - Any Python code! |
158 | | - |
159 | | - description = f"Analysis of {self.inputs.city}" |
160 | | - features = ["culture", "economy", "lifestyle"] |
161 | | - score = 8.5 |
162 | | - |
163 | | - return self.Outputs( |
164 | | - description=description, |
165 | | - key_features=json.dumps(features), |
166 | | - score=str(score) |
167 | | - ) |
168 | | - ``` |
169 | | - |
170 | | - |
171 | | - |
172 | | -### Step 3: Create and Start the Runtime |
173 | | - |
174 | | -Create the runtime and register your nodes: |
175 | | - ```python |
176 | | - from exospherehost import Runtime |
177 | | - |
178 | | - # Initialize the runtime with your nodes |
179 | | - runtime = Runtime( |
180 | | - name="city-analysis-runtime", |
181 | | - namespace="my-project", |
182 | | - nodes=[CityAnalyzerNode] |
183 | | - ) |
184 | | - |
185 | | - # Start the runtime (this will block the main thread) |
186 | | - runtime.start() |
187 | | - ``` |
188 | | - |
189 | | - **Run your application:** |
190 | | - ```bash |
191 | | - uv run main.py |
192 | | - ``` |
193 | | - |
194 | | - Your runtime is now running and ready to process workflows! 🎉 |
195 | | - |
196 | | -### Step 4: Define Your First Graph |
197 | | - |
198 | | - Graphs can be defined using JSON objects or with the new model-based Python SDK (beta) for better type safety and validation. See [Graph definitions](https://docs.exosphere.host/exosphere/create-graph/) for more examples. |
199 | | - |
200 | | - |
201 | | - |
202 | | - ```python |
203 | | - from exospherehost import StateManager, GraphNodeModel, RetryPolicyModel, RetryStrategyEnum |
204 | | - |
205 | | - async def create_graph(): |
206 | | - state_manager = StateManager(namespace="hello-world") |
207 | | - |
208 | | - graph_nodes = [ |
209 | | - GraphNodeModel( |
210 | | - node_name="MyFirstNode", |
211 | | - namespace="hello-world", |
212 | | - identifier="describe_city", |
213 | | - inputs={"city": "initial"}, |
214 | | - next_nodes=[] |
215 | | - ) |
216 | | - ] |
217 | | - |
218 | | - # Optional: Define retry policy (beta) |
219 | | - retry_policy = RetryPolicyModel( |
220 | | - max_retries=3, |
221 | | - strategy=RetryStrategyEnum.EXPONENTIAL, |
222 | | - backoff_factor=2000 |
223 | | - ) |
224 | | - |
225 | | - # Create graph with model-based approach (beta) |
226 | | - result = await state_manager.upsert_graph( |
227 | | - graph_name="my-first-graph", |
228 | | - graph_nodes=graph_nodes, |
229 | | - secrets={}, |
230 | | - retry_policy=retry_policy # beta |
231 | | - ) |
232 | | - ``` |
233 | | - |
234 | | -## Quick Start with Docker Compose |
235 | | - |
236 | | -Get Exosphere running locally in under 2 minutes: |
237 | | - |
238 | | -```bash |
239 | | -# Option 1: With cloud MongoDB (recommended) |
240 | | -echo "MONGO_URI=your-mongodb-connection-string" > .env |
241 | | -curl -O https://raw.githubusercontent.com/exospherehost/exospherehost/main/docker-compose/docker-compose.yml |
242 | | -docker compose up -d |
243 | | - |
244 | | -# Option 2: With local MongoDB (development) |
245 | | -curl -O https://raw.githubusercontent.com/exospherehost/exospherehost/main/docker-compose/docker-compose-with-mongodb.yml |
246 | | -docker compose -f docker-compose-with-mongodb.yml up -d |
247 | | -``` |
248 | | - |
249 | | -**Environment Configuration:** |
250 | | -- Docker Compose automatically loads `.env` files from the working directory |
251 | | -- Create your `.env` file in the same directory as your docker-compose file |
252 | | - |
253 | | -Access your services: |
254 | | - |
255 | | -- **Dashboard**: `http://localhost:3000` |
256 | | -- **API**: `http://localhost:8000` |
257 | | - |
258 | | -> **📝 Note**: This configuration is for **development and testing only**. For production deployments, environment variable customization, and advanced configuration options, please read the complete **[Docker Compose Setup Guide](https://docs.exosphere.host/docker-compose-setup)**. |
259 | | -
|
260 | | -## 📚 Documentation & Resources |
| 72 | +| Concept | Description | |
| 73 | +|---------|-------------| |
| 74 | +| **Fanout** | Distribute work across multiple parallel instances of a node | |
| 75 | +| **Unite** | Combine results from multiple parallel executions | |
| 76 | +| **Signals** | Inter-node communication and event handling | |
| 77 | +| **Retry Policy** | Configurable failure handling and recovery | |
| 78 | +| **Store** | Persistent storage for workflow state and data | |
| 79 | +| **Triggers** | Automatic scheduling with cron expressions | |
261 | 80 |
|
262 | | -### **Essential Guides** |
263 | | -- **[Getting Started Guide](https://docs.exosphere.host/getting-started)**: Complete walkthrough for new users |
264 | | -- **[Docker Compose Setup](https://docs.exosphere.host/docker-compose-setup)**: Run Exosphere locally in minutes |
265 | | -- **[Architecture Guide](https://docs.exosphere.host/exosphere/architecture)**: Understand core concepts like fanout and unite |
266 | | -- **[Youtube Walkthroughs](https://www.youtube.com/@exospherehost)**: Step by step demos on Exosphere and how to build reliable flows with sample code. |
267 | | -- **[Featured Exosphere Projects](https://github.com/exospherehost/exosphereprojects)**: Templates on common projects on Exosphere, pull and run! |
| 81 | +--- |
268 | 82 |
|
269 | | -### **Advanced Topics** |
270 | | -- **[State Manager Setup](https://docs.exosphere.host/exosphere/state-manager-setup)**: Production deployment guide |
271 | | -- **[Dashboard Guide](https://docs.exosphere.host/exosphere/dashboard)**: Visual workflow management |
272 | | -- **[Graph Definitions](https://docs.exosphere.host/exosphere/create-graph/)**: Building complex workflows |
| 83 | +## Resources |
273 | 84 |
|
274 | | -### **Community & Support** |
275 | | -- **[Official Documentation](https://docs.exosphere.host)**: Complete reference and tutorials |
276 | | -- **[Discord Community](https://discord.com/invite/zT92CAgvkj)**: Get help and connect with other developers |
277 | | -- **[GitHub Issues](https://github.com/exospherehost/exospherehost/issues)**: Report bugs and request features |
278 | | -- **[PyPI Package](https://pypi.org/project/exospherehost/)**: Latest stable releases |
| 85 | +| Resource | Notes | |
| 86 | +|----------|-------------| |
| 87 | +| [**Getting Started Guide**](https://docs.exosphere.host/getting-started) | Complete walkthrough for new users | |
| 88 | +| [**Docker Compose Setup**](https://docs.exosphere.host/docker-compose-setup) | Run Exosphere locally in minutes | |
| 89 | +| [**Architecture Guide**](https://docs.exosphere.host/exosphere/architecture) | Understand core concepts like fanout and unite | |
| 90 | +| [**YouTube Walkthroughs**](https://www.youtube.com/@exospherehost) | Step-by-step demos on Exosphere with sample code | |
| 91 | +| [**Featured Projects**](https://github.com/exospherehost/exosphereprojects) | Templates on common projects, pull and run | |
279 | 92 |
|
280 | 93 |
|
| 94 | +### Community and Support |
281 | 95 |
|
| 96 | +| Resource | Description | |
| 97 | +|----------|-------------| |
| 98 | +| [**Official Documentation**](https://docs.exosphere.host) | Complete reference and tutorials | |
| 99 | +| [**Discord Community**](https://discord.com/invite/zT92CAgvkj) | Get help and connect with other developers | |
| 100 | +| [**GitHub Issues**](https://github.com/exospherehost/exospherehost/issues) | Report bugs and request features | |
| 101 | +| [**PyPI Package**](https://pypi.org/project/exospherehost/) | Latest stable releases | |
282 | 102 |
|
283 | | -## 🌟 Open Source Commitment |
| 103 | +--- |
284 | 104 |
|
285 | | -We believe that open source is the foundation of innovation and progress. Exosphere is our contribution to this movement, and we're committed to supporting the community in multiple ways: |
| 105 | +## Open Source Commitment |
286 | 106 |
|
287 | | -### **Our Promise to the Community** |
| 107 | +We believe that humanity would not have been able to achieve the level of innovation and progress we have today without the support of open source and community. We want to be a part of this movement. |
288 | 108 |
|
289 | | -1. **🔄 Open Source First**: The majority of our codebase is open source and available to everyone |
290 | | -2. **💰 Giving Back**: A portion of our profits goes directly to supporting open source projects and communities |
291 | | -3. **🎓 Mentorship**: We actively collaborate with student programs to mentor the next generation of developers |
292 | | -4. **🤝 Community Driven**: We welcome contributions, feedback, and collaboration from developers worldwide |
| 109 | +Please feel free to reach out to us at [nivedit@exosphere.host](mailto:nivedit@exosphere.host). Let's push the boundaries of possibilities for humanity together. |
293 | 110 |
|
294 | | -### **Get Involved** |
| 111 | +--- |
295 | 112 |
|
296 | | -- **Contributors**: Help us build the future of AI infrastructure |
297 | | -- **Users**: Your feedback shapes our roadmap and priorities |
298 | | -- **Students**: Join our mentorship programs and grow your skills |
299 | | -- **Organizations**: Partner with us to advance open source AI tools |
| 113 | +## Release Cycle and Roadmap |
300 | 114 |
|
301 | | -**Ready to make a difference?** Reach out to us at [nivedit@exosphere.host](mailto:nivedit@exosphere.host) and let's push the boundaries of what's possible together! 🚀 |
302 | | - |
303 | | -## 🎯 Ready to Get Started? |
| 115 | +Exosphere follows a predictable, calendar-based release process: |
304 | 116 |
|
305 | | -Exosphere is designed to make AI workflow development accessible, scalable, and production-ready. Whether you're building your first AI agent or scaling to thousands of parallel workflows, Exosphere provides the infrastructure you need. |
306 | | - |
307 | | -### **Next Steps:** |
308 | | -1. **⭐ Star this repository** to show your support |
309 | | -2. **🚀 Try the quick start** with our Docker Compose setup |
310 | | -3. **💬 Join our Discord** community for help and discussions |
311 | | -4. **📖 Read the docs** for comprehensive guides and examples |
312 | | -5. **🤝 Contribute** to help us build the future of AI infrastructure |
| 117 | +| Aspect | Description | |
| 118 | +|--------|-------------| |
| 119 | +| **Monthly Releases** | A new stable version ships at the end of every month | |
| 120 | +| **Issue and PR Labelling** | Work intended for a release is tagged `YYYY:Mon` (e.g., `2026:Jan`). Filter by this label in GitHub to see exactly what is planned | |
| 121 | +| **Living Roadmap** | The collection of items carrying the current month's label is our public roadmap. Follow along in GitHub Projects to track progress in real time | |
313 | 122 |
|
314 | 123 | --- |
315 | 124 |
|
316 | 125 | ## Contributing |
317 | 126 |
|
318 | 127 | We welcome community contributions. For guidelines, refer to our [CONTRIBUTING.md](https://github.com/exospherehost/exospherehost/blob/main/CONTRIBUTING.md). |
319 | 128 |
|
320 | | - |
321 | | - |
322 | | - |
| 129 | +Thanks to our awesome contributors! |
323 | 130 |
|
| 131 | + |
0 commit comments