Agentic Orchestration System

A simple agentic framework for constructing software, doing research, and managing data using Claude Code.

Overview

This system provides a complete file-based orchestration system for managing Claude Code jobs:

• File-based messaging bus (inbox/outbox) with UUID correlation
• Orchestrator daemon for routing commands and creating jobs
• Queue managers for executing jobs across multiple compute machines
• Worker harness for running Claude Code jobs
• JSON schema validation for all data structures
• Complete job lifecycle management (queued → running → finished/failed)

Installation

# Clone the repository
git clone https://github.com/cshenry/AgenticOrchestrationSystem
cd AgenticOrchestrationSystem

# Install dependencies
pip install -r requirements.txt

# Install the package in development mode
pip install -e .

Quick Start

1. Initial Setup

The system uses Dropbox for file synchronization. Set up the AgentStore directory:

# The AgentStore directory structure is created automatically
# Default location: ~/Dropbox/AgentStore/

# Copy example config
cp config.example.yaml ~/Dropbox/AgentStore/config.yaml

# Edit config if needed
vim ~/Dropbox/AgentStore/config.yaml

2. Create Your First User

Create a user file in ~/Dropbox/AgentStore/users/<username>.json:

{
  "id": "yourusername",
  "slack_username": "yourusername",
  "name": "Your Name",
  "email": "your.email@example.com",
  "permissions": "admin"
}

3. Log In

orchestrator login yourusername

4. Create a Project

Create a project in ~/Dropbox/AgentStore/projects/YourProject/project.json:

{
  "id": "YourProject",
  "directory": "/path/to/your/project",
  "git_repository": "https://github.com/you/yourproject",
  "create_time": "2025-10-27T00:00:00Z",
  "update_time": "2025-10-27T00:00:00Z",
  "description": "Your project description",
  "owner": "yourusername",
  "collaborators": [],
  "code_research_document": "/path/to/your/project/DOCS.md",
  "default_queue": "poplar"
}

5. Create a Queue

Create a queue in ~/Dropbox/AgentStore/Queues/yourqueue/queue.json:

{
  "config": {
    "id": "yourqueue",
    "worker_count": 5,
    "claude_cli_path": "claude",
    "rest_interval": 60,
    "claude_default_args": ["-p"],
    "max_job_runtime_seconds": 3600
  },
  "runtime": {
    "process_id": null,
    "active_jobs": 0,
    "last_update": null,
    "queued_jobs": 0
  }
}

Also create the job directories:

mkdir -p ~/Dropbox/AgentStore/Queues/yourqueue/Jobs/{queued_jobs,running_jobs,finished_jobs,failed_jobs}

6. Start the System

# Start orchestrator (in one terminal)
PYTHONPATH=. python3 -m src.orchestrator.orchestrator

# Start queue manager (in another terminal)
PYTHONPATH=. python3 -m src.queue_manager.queue_manager yourqueue

Or use the CLI commands:

PYTHONPATH=. python3 -m src.cli.cli orchestrator-daemon start
PYTHONPATH=. python3 -m src.cli.cli queue-manager --queue yourqueue

7. Submit Your First Job

# Set current project
PYTHONPATH=. python3 -m src.cli.cli set-project YourProject

# Submit a code research job
PYTHONPATH=. python3 -m src.cli.cli research-code --project YourProject

CLI Commands

State Management

• login <username> - Set username for CLI
• set-project <project-id> - Set current project
• set-prd <prd-id> - Set current PRD

Browsing Data

• list-users - List all users
• list-projects - List all projects
• list-prds [--project PROJECT] - List PRDs
• list-jobs [--status STATUS] [--project PROJECT] - List jobs
• list-queues - List all queues

Viewing Data

• view-user <user-id> - View user details
• view-project <project-id> - View project details
• view-prd <prd-id> - View PRD details
• view-job <job-id> - View job details
• view-queue <queue-name> - View queue status

Agent Commands (Create Jobs)

• research-code --project PROJECT [--queue QUEUE] [--follow SECONDS] - Research codebase
• create-prd --project PROJECT [--queue QUEUE] - Create PRD
• create-tasks --project PROJECT --prd PRD [--queue QUEUE] - Generate tasks from PRD
• implement-tasks --project PROJECT --prd PRD [--queue QUEUE] - Implement tasks
• create-environment --project PROJECT - Create venv using venvman

Daemon Control

• orchestrator-daemon {start|stop|status} - Control orchestrator daemon
• queue-manager --queue QUEUE - Start queue manager

Global Flags

• --format {json|text} - Output format (default: text)
• --verbose - Show debug output
• --follow SECONDS - Wait for response (for agent commands)

Directory Structure

~/Dropbox/AgentStore/
├── config.yaml                    # Global configuration
├── messaging/
│   ├── inbox/                     # Incoming messages
│   ├── processed_inbox/           # Processed messages
│   ├── outbox/                    # Outgoing responses
│   └── processed_outbox/          # Sent responses
├── users/
│   └── <user-id>.json             # User definitions
├── projects/
│   └── <project-name>/
│       ├── project.json           # Project configuration
│       ├── queued_prds/           # Queued PRDs
│       ├── running_prds/          # Running PRDs
│       └── finished_prds/         # Completed PRDs
├── Queues/
│   └── <queue-name>/
│       ├── queue.json             # Queue configuration
│       ├── log.out                # Queue logs
│       └── Jobs/
│           ├── queued_jobs/       # Jobs waiting to run
│           ├── running_jobs/      # Currently running jobs
│           ├── finished_jobs/     # Completed jobs
│           └── failed_jobs/       # Failed jobs
├── JSONValidation/
│   ├── Schemas/                   # JSON schemas
│   │   ├── user.schema.json
│   │   ├── message.schema.json
│   │   ├── project.schema.json
│   │   ├── prd.schema.json
│   │   ├── queue.schema.json
│   │   └── job.schema.json
│   └── errors.out                 # Validation errors
└── logs/
    ├── orchestrator.log           # Orchestrator logs
    ├── queue-<queue>.log          # Queue manager logs
    └── events-YYYY-MM.jsonl       # Event logs

Architecture

Components

1. Orchestrator: Routes messages, creates jobs, handles job completion
2. Queue Manager: Monitors jobs, starts workers, enforces limits
3. Worker Harness: Executes Claude Code jobs, captures output
4. CLI: User interface for all operations
5. JSON Validator: Validates all data structures

Job Lifecycle

1. User submits command via CLI → writes message to inbox
2. Orchestrator processes message → creates job in queue
3. Queue Manager detects job → starts worker harness
4. Worker Harness executes Claude → captures output
5. Job moves to finished_jobs → Orchestrator handles completion
6. Project/PRD updated with results

Message Flow

User → CLI → inbox → Orchestrator → outbox → CLI → User
                ↓
              Job Creation
                ↓
        Queue Manager → Worker Harness → Claude Code
                ↓
          finished_jobs
                ↓
         Orchestrator (completion handler)

Development

Running Tests

# Run all tests
pytest tests/

# Run specific test file
pytest tests/test_validator.py

# Run with verbose output
pytest -v tests/

Manual Testing

The system includes test data:

• Test user: chenry
• Test project: TestProject
• Test queue: poplar

Code Structure

src/
├── validator/          # JSON schema validation
│   ├── __init__.py
│   └── json_validator.py
├── common/             # Shared utilities
│   ├── __init__.py
│   ├── utils.py
│   └── config.py
├── cli/                # Command-line interface
│   ├── __init__.py
│   ├── cli.py
│   └── commands.py
├── orchestrator/       # Orchestrator daemon
│   ├── __init__.py
│   ├── orchestrator.py
│   ├── handlers.py
│   └── job_manager.py
├── queue_manager/      # Queue manager daemon
│   ├── __init__.py
│   └── queue_manager.py
└── worker/             # Worker harness
    ├── __init__.py
    └── worker_harness.py

Troubleshooting

Orchestrator Won't Start

• Check if already running: orchestrator orchestrator-daemon status
• Check lock file: ~/Dropbox/AgentStore/.orchestrator.lock
• Check logs: ~/Dropbox/AgentStore/logs/orchestrator.log

Jobs Not Running

• Check queue manager is running
• Check queue.json runtime.active_jobs
• Check worker_count limit
• Check job files in queued_jobs directory
• Check logs: ~/Dropbox/AgentStore/logs/queue-<queue>.log

Jobs Failing

• Check job error in runtime.error field
• Verify Claude command exists in ~/.claude/commands/
• Check working directory exists
• Check logs for detailed error messages

Permission Errors

• Verify user.json has correct permissions field
• Check file ownership in AgentStore directory
• Verify Dropbox sync is working

Future Enhancements (v0.1+)

• Job priority and urgency
• Auto-retry failed jobs
• Schema migration tools
• Job cancellation
• More Claude commands
• Slack integration (v1.0)
• Multi-step workflows (v1.0)
• Web dashboard (v1.0)

Documentation

• PRD_Agentic_Framework_v0.5_FINAL.md - Complete Product Requirements Document
• InitialUserStory.md - Initial user story and requirements
• tasks/tasks-PRD_Agentic_Framework_v0.5_FINAL.md - Implementation task list

Contributing

This is currently a personal project. For bugs or feature requests, please create an issue.

License

MIT