Streamline Beads backlog workflow

.beads/.gitignore

@@ -0,0 +1,2 @@
+*.db
+*.db-*

AGENTS.md

@@ -1,20 +1,101 @@
 # Agent Instructions
 
-## Documentation Workflow
-- After each batch of changes, add a `CHANGELOG.md` entry with an ISO 8601 date/time stamp in United States Eastern time (include the timezone code, e.g., `America/New_York` or `ET`) and developer-facing detail (files, modules, functions, variables, and rationale). Every commit should correspond to a fresh entry.
-- Maintain `README.md` as the canonical description of the project; update it whenever behaviour or workflows change. Archive older versions separately when requested.
-- Keep the `docs/` wiki and provisioning guides (`SETUP.md`, `ENVIRONMENT_NEEDS.md`) in sync with code updates; add or revise the
-  relevant page whenever features, modules, or workflows change.
-- After each iteration, refresh `ISSUES.md`, `SOT.md`, `PLAN.md`, `RECOMMENDATIONS.md`, `TODO.md`, and related documentation to stay in sync with the codebase.
-- Ensure `TODO.md` retains the `Completed`, `Priority Tasks`, and `Recommended Waiting for Approval Tasks` sections, moving finished items under `Completed` at the end of every turn.
-- Make every task in `TODO.md` atomic: each entry must describe a single, self-contained deliverable with enough detail to execute and verify without cross-referencing additional context.
-- Update `RESUME_NOTES.md` at the end of every turn so the next session starts with accurate context.
-- When beginning a turn, review `README.md`, `PROJECT.md`, `PLAN.md`, `RECOMMENDATIONS.md`, `ISSUES.md`, `SOT.md`, `ROADMAP.md`, `PLANNING_THOUGHTS.md`, and the `research/` wiki to harvest new actionable work. Maintain at least ten quantifiable, prioritised items in the `Priority Tasks` section of `TODO.md`, adding context or links when needed.
-- Keep `ROADMAP.md`, `PLANNING_THOUGHTS.md`, and the `research/` docs aligned with each iteration when new discoveries or decisions occur.
-- After completing any task, immediately update `TODO.md`, check for the next actionable item, and continue iterating until all unblocked `Priority Tasks` are exhausted for the session.
-- Continuously loop through planning and execution: finish a task, document it, surface new follow-ups, and resume implementation so long as environment blockers allow. If extra guidance would improve throughput, extend these instructions proactively.
-
-## Style Guidelines
-- Use descriptive Markdown headings starting at level 1 for top-level documents.
-- Keep lines to 120 characters or fewer when practical.
-- Prefer bullet lists for enumerations instead of inline commas.
+## Beads Issue Tracking
+- Use the `bd` Beads CLI for all issue, task, and dependency tracking in this repository.
+- Keep every issue up to date: assign `codex` when the work is agent-owned, `human` when it requires external help, and maintain `area/*`, `priority/*`, and `status/blocked` labels so filtering remains useful.
+- Immediately capture newly discovered work or blockers in Beads before continuing implementation, and wire dependencies (`bd dep add`) so `bd ready` only lists actionable tasks.
+- If the `.beads/` database is missing, run `bd init` (and `bd import -i .beads/issues.jsonl` if needed) and commit the resulting files.
+
+## Iteration Workflow
+- Begin each session with `bd ready`/`bd list --status open` to understand available work and outstanding blockers.
+- Close or update issues as soon as changes land, and keep dependency graphs accurate so downstream tasks stay blocked until prerequisites resolve.
+- Add an entry to `CHANGELOG.md` for every change set with an ISO 8601 timestamp in America/New_York summarizing developer-facing impact and rationale.
+- Rely on Beads for planning instead of sprawling Markdown checklists; use docs only for long-lived guidance that end users need.
+
+## Documentation & Communication
+- Update `README.md`, `docs/`, and provisioning guides only when the implementation meaningfully changes behaviour or workflows touched by the turn.
+- Keep high-level orientation docs (`ISSUES.md`, `PLAN.md`, `ROADMAP.md`, `RECOMMENDATIONS.md`) short and reference the relevant Beads issue IDs instead of duplicating task lists.
+- Maintain `RESUME_NOTES.md` at the end of each turn so the next session starts with accurate context.
+
+## bd quickstart reference output
+```
+bd - Dependency-Aware Issue Tracker
+
+Issues chained together like beads.
+
+GETTING STARTED
+  bd init   Initialize bd in your project
+            Creates .beads/ directory with project-specific database
+            Auto-detects prefix from directory name (e.g., myapp-1, myapp-2)
+
+  bd init --prefix api   Initialize with custom prefix
+            Issues will be named: api-1, api-2, ...
+
+CREATING ISSUES
+  bd create "Fix login bug"
+  bd create "Add auth" -p 0 -t feature
+  bd create "Write tests" -d "Unit tests for auth" --assignee alice
+
+VIEWING ISSUES
+  bd list       List all issues
+  bd list --status open  List by status
+  bd list --priority 0  List by priority (0-4, 0=highest)
+  bd show bd-1       Show issue details
+
+MANAGING DEPENDENCIES
+  bd dep add bd-1 bd-2     Add dependency (bd-2 blocks bd-1)
+  bd dep tree bd-1  Visualize dependency tree
+  bd dep cycles      Detect circular dependencies
+
+DEPENDENCY TYPES
+  blocks  Task B must complete before task A
+  related  Soft connection, doesn't block progress
+  parent-child  Epic/subtask hierarchical relationship
+  discovered-from  Auto-created when AI discovers related work
+
+READY WORK
+  bd ready       Show issues ready to work on
+            Ready = status is 'open' AND no blocking dependencies
+            Perfect for agents to claim next work!
+
+UPDATING ISSUES
+  bd update bd-1 --status in_progress
+  bd update bd-1 --priority 0
+  bd update bd-1 --assignee bob
+
+CLOSING ISSUES
+  bd close bd-1
+  bd close bd-2 bd-3 --reason "Fixed in PR #42"
+
+DATABASE LOCATION
+  bd automatically discovers your database:
+    1. --db /path/to/db.db flag
+    2. $BEADS_DB environment variable
+    3. .beads/*.db in current directory or ancestors
+    4. ~/.beads/default.db as fallback
+
+AGENT INTEGRATION
+  bd is designed for AI-supervised workflows:
+    • Agents create issues when discovering new work
+    • bd ready shows unblocked work ready to claim
+    • Use --json flags for programmatic parsing
+    • Dependencies prevent agents from duplicating effort
+
+DATABASE EXTENSION
+  Applications can extend bd's SQLite database:
+    • Add your own tables (e.g., myapp_executions)
+    • Join with issues table for powerful queries
+    • See database extension docs for integration patterns:
+      https://github.com/steveyegge/beads/blob/main/EXTENDING.md
+
+GIT WORKFLOW (AUTO-SYNC)
+  bd automatically keeps git in sync:
+    • ✓ Export to JSONL after CRUD operations (5s debounce)
+    • ✓ Import from JSONL when newer than DB (after git pull)
+    • ✓ Works seamlessly across machines and team members
+    • No manual export/import needed!
+  Disable with: --no-auto-flush or --no-auto-import
+
+Ready to start!
+Run bd create "My first issue" to create your first issue.
+```

CHANGELOG.md

@@ -1,5 +1,20 @@
 # Changelog
 
+## [2025-10-20T04:55:28-04:00 (America/New_York)]
+### Changed
+- Created staging and dataset blocker issues (`meshmind-95`–`meshmind-103`) and wired dependencies so `bd ready` surfaces only actionable work.
+- Assigned `codex`/`human` ownership and `area/*` + `status/blocked` labels across the Beads database to enable filtering.
+- Streamlined AGENTS.md and ISSUES.md to point back to the tracker while highlighting the new blocker queue.
+
+## [2025-10-20T04:12:44-04:00 (America/New_York)]
+### Added
+- Introduced `.beads/` project database and exported Beads issue JSONL by initializing bd and migrating existing TODO items.
+- Added `reference-docs/` snapshots of upstream Beads documentation (`Beads-*.md`) for local reference.
+
+### Changed
+- Rewrote `AGENTS.md` to mandate Beads-based workflow and appended the `bd quickstart` reference output; simplified `TODO.md` to delegate planning to bd.
+- Imported every task from the legacy `TODO.md` into the Beads tracker with appropriate open/closed status and added an `ISSUES.md` note pointing contributors to bd commands.
+
 ## [2025-10-16T20:39:06-04:00 (America/New_York)]
 ### Added
 - Added live integration coverage for Memgraph, Neo4j, and Redis via `meshmind/tests/test_integration_live.py` and configured

ISSUES.md

@@ -1,41 +1,34 @@
 # Issues Checklist
 
+> [!NOTE]
+> Issue and task tracking lives in the Beads tracker (`bd`). Use this page as a quick orientation for the highest-impact items and check `bd show <issue>` for full details.
+
 ## Blockers
-- [x] MeshMind client fails without the `mgclient` module; introduce lazy driver initialization or documented in-memory fallback.
-- [x] Register a default embedding encoder (OpenAI or sentence-transformers) during startup so extraction and hybrid search can run.
-- [x] Update OpenAI integration to match the current SDK (Responses API payload, embeddings API response structure).
-- [x] Replace eager `tiktoken` imports in `meshmind.core.utils` and `meshmind.pipeline.compress` with guarded, optional imports.
-- [x] Align declared Python requirement with supported dependencies (project now pins Python >=3.11,<3.13).
+- [ ] `meshmind-95` – Provision a Celery worker + Redis staging stack so maintenance and integration tests can hit real services.
+- [ ] `meshmind-96` – Deploy an authenticated REST/gRPC staging environment to validate curl/grpcurl documentation.
+- [ ] `meshmind-98` – Stand up long-lived Memgraph/Neo4j clusters dedicated to benchmarking.
+- [ ] `meshmind-99` – Regenerate and publish a ≥10k-memory synthetic dataset for consolidation benchmarks.
+- [ ] `meshmind-101` – Restore outbound PyPI (or provide a mirror) so dependency locks can be regenerated.
 
-- [ ] Maintain pip/uv package download access (confirmed working on 2025-10-15) so dependency lock regeneration can proceed reliably across sessions.
-## High Priority
-- [x] Provide configuration documentation and examples for Memgraph, Redis, and OpenAI environment variables.
-- [x] Add automated tests or smoke checks that run without external services (mock OpenAI, stub Memgraph driver).
-- [x] Create real docker-compose services for Memgraph and Redis or remove the placeholder file.
-- [x] Centralize LLM provider usage behind a configurable client wrapper to remove direct `openai` imports scattered through the codebase.
-- [x] Surface LLM override fields via REST/gRPC payloads and integration tests so service clients can select providers/models like the CLI.
-- [x] Document Neo4j driver requirements and verify connectivity against a live cluster (integration suite now hits the docker-compose Neo4j service).
-- [ ] Exercise the new namespace/entity-label filtering against live Memgraph/Neo4j datasets to confirm Cypher predicates behave as expected.
-- [x] Regenerate `uv.lock` to reflect the updated dependency set (`pymgclient`, `fastapi`, `uvicorn`, extras) so CI tooling stays in sync.
-## Medium Priority
-- [x] Persist results from consolidation and compression tasks back to the database (currently in-memory only).
-- [x] Refine `Memory.importance` scoring to reflect actual ranking heuristics instead of a constant.
-- [x] Add vector, regex, and exact-match search helpers to match stated feature set or update documentation to demote them.
-- [x] Harden Celery tasks to initialize dependencies lazily and log failures when the driver is unavailable.
-- [x] Validate consolidation heuristics on larger datasets to measure ranking accuracy and resource usage (synthetic fixtures and benchmark scripts cover scale; rerun with production datasets when available).
-- [x] Document and implement a conflict-resolution/backoff strategy for consolidation when merged metadata conflicts (configurable via `MAINTENANCE_MAX_ATTEMPTS` and `MAINTENANCE_BASE_DELAY_SECONDS`).
-- [x] Revisit the compatibility shim once production environments support Pydantic 2.x so the real models can be restored (shim removed; native Pydantic models now required).
-- [x] Replace the gRPC dataclass shim with generated protobuf definitions; follow-up integration tests remain pending until a real gRPC server is provisioned.
-- [x] Implement a production-ready gRPC server (leveraging the generated protobuf modules). Async helpers now live in `meshmind.api.grpc_server`; integration tests remain blocked on staging infrastructure.
-- [ ] Add end-to-end gRPC integration tests (deploy server + grpcurl smoke) once staging infrastructure is provisioned.
-- [ ] Push graph-backed retrieval into Memgraph/Neo4j search capabilities once available (current wrappers now filter/paginate server-side but still score vectors in Python).
-- [ ] Reconcile tests that depend on `Memory.pre_init` and outdated OpenAI interfaces with the current implementation.
-- [x] Expose `memory_counts` via a gRPC endpoint to keep service interfaces aligned.
-- [x] Add linting, formatting, and type-checking tooling to improve code quality.
+## High Priority (next once blockers clear)
+- [ ] `meshmind-82` – Implement backend-native vector similarity queries for Memgraph/Neo4j.
+- [ ] `meshmind-83` – Run `scripts/consolidation_benchmark.py` with the refreshed dataset and document retry defaults.
+- [ ] `meshmind-84` – Execute pagination benchmarks against the staging clusters and update retrieval guidance.
+- [ ] `meshmind-85` – Add Celery/Redis-backed integration tests for maintenance retry flags.
+- [ ] `meshmind-86` – Re-run curl/grpcurl documentation steps against the deployed staging endpoints.
+- [ ] `meshmind-87` – Expand gRPC ingestion/search integration coverage once infrastructure lands.
+- [ ] `meshmind-88` – Publish protobuf client artifacts after the release pipeline exists.
+- [ ] `meshmind-89` – Automate the live integration suite in CI once maintenance tests pass.
+- [ ] `meshmind-90` – Document ingestion workflows for the synthetic dataset generator after benchmarking.
+- [ ] `meshmind-91` – Capture release notes for retiring REST/Celery shims once migration source material is ready.
+- [ ] `meshmind-92` – Write gRPC CLI usage examples after the smoke tests complete.
+- [ ] `meshmind-93` – Identify observability exporters and outline integration steps.
+- [ ] `meshmind-94` – Explore UI concepts after the API hardening backlog (`meshmind-103`) lands.
+- [ ] `meshmind-97` – Stand up the protobuf artifact release pipeline.
+- [ ] `meshmind-100` – Gather authoritative REST/Celery retirement context for documentation.
+- [ ] `meshmind-102` – Track upstream Pydantic packaging so the compatibility shim can be retired for good.
+- [ ] `meshmind-103` – Finish the API hardening backlog to unblock downstream UX work.
 
-- [ ] Validate the new Docker Compose stacks (root and `meshmind/tests/docker/`) on an environment with container support and document host requirements (ports, resources).
-## Low Priority / Nice to Have
-- [x] Offer alternative storage backends (in-memory driver, SQLite, etc.) for easier local development.
-- [x] Provide an administrative dashboard or CLI commands for listing namespaces, counts, and maintenance statistics (CLI admin subcommands now expose predicates, telemetry, and graph checks).
-- [ ] Publish onboarding guides and troubleshooting FAQs for contributors.
-- [ ] Explore plugin registration for embeddings and retrieval strategies to reduce manual wiring.
+## Reference
+- Historical items completed prior to the Beads migration remain closed in the tracker (`bd list --status closed`).
+- Use `bd ready` for the actionable queue after the blockers above are addressed.