Expand roadmap tasks for parity gaps

CHANGELOG.md

@@ -1,5 +1,19 @@
 # Changelog
 
+## [2025-10-15T17:30:19-04:00 (America/New_York)]
+### Updated
+- Expanded `ROADMAP_TASKS.md` with MCP search/graph tooling, chat-store compatibility, shared crew scopes, conversation summary
+  memory, and multi-embedder routing tasks derived from competitor research so feature coverage matches the prioritized roadmap.
+
+## [2025-10-15T01:13:29-04:00 (America/New_York)]
+### Added
+- Authored `GOALS.md`, `PLANNING_THOUGHTS.md`, and `ROADMAP.md` to capture competitor-informed strategic goals, planning
+  options, and a prioritized feature roadmap aligning MeshMind with Mem0, Graphiti, and Zep capabilities.
+
+### Updated
+- Refreshed planning collateral (`PLAN.md`, `SOT.md`, `RECOMMENDATIONS.md`, `TODO.md`, `ISSUES.md`, `RESUME_NOTES.md`) to
+  reference the new strategic documents and surface roadmap-aligned backlog items.
+
 ## [2025-10-14T22:51:20-04:00 (America/New_York)]
 ### Changed
 - Documented LLM override precedence in `README.md`, expanded service documentation in `docs/api.md` and

GOALS.md

@@ -0,0 +1,36 @@
+# MeshMind Strategic Goals
+
+## Platform and Interface Parity
+- Deliver an official MeshMind MCP server with tooling coverage for memory CRUD, fact search, triplet queries, and graph cleanup so agent runtimes can treat MeshMind as a drop-in replacement for Mem0, Graphiti, and Zep offerings.【F:research/meshmind_exceed_recommendations.md†L4-L9】【F:research/ai_memory_features_catalog.csv†L2-L27】
+- Publish stable REST, gRPC, and MCP contracts accompanied by SDKs (Python, TypeScript) and example integrations for LangGraph, CrewAI, and other orchestration frameworks.【F:research/ai_memory_features_catalog.csv†L21-L59】
+- Provide command-line tooling that mirrors hosted competitors by covering provisioning, maintenance, reindexing, export/import, and evaluation flows.【F:research/meshmind_exceed_recommendations.md†L16-L23】
+
+## Graph Excellence and Temporal Intelligence
+- Add bi-temporal edge modeling (valid/transaction windows) with `as_of` querying semantics, invalidation hooks, and historical replay utilities to match Graphiti/Zep capabilities.【F:research/meshmind_exceed_recommendations.md†L5-L9】【F:research/meshmind_gap_table.csv†L2-L5】
+- Implement node-distance and focal-entity rerankers alongside multi-hop traversal recipes that combine BM25, embeddings, RRF/MMR, and BFS prioritization.【F:research/meshmind_exceed_recommendations.md†L5-L9】【F:research/meshmind_gap_table.csv†L6-L12】
+- Support graph-personalization features such as persona nodes, implicit preference graphs, and configurable rerank boosts for first-party experiences.【F:research/meshmind_exceed_recommendations.md†L24-L27】
+
+## Scope, Tenancy, and Governance
+- Introduce multi-level scoping primitives for user, agent, session, and run identifiers across storage, retrieval, APIs, and tooling so MeshMind can mirror Mem0 and Zep tenancy models.【F:research/meshmind_gap_table.csv†L3-L4】【F:research/ai_memory_features_catalog.csv†L3-L33】
+- Harden auth with API keys/JWT, per-scope quotas, and tenant isolation checks that execute in graph drivers and service layers by default.【F:research/meshmind_exceed_recommendations.md†L9-L15】【F:research/ai_memory_features_catalog.csv†L20-L59】
+- Ship data-governance controls including PII detection, redaction, retention policies, and encryption-at-rest toggles with audit reporting for compliance-sensitive deployments.【F:research/meshmind_exceed_recommendations.md†L27-L32】【F:research/meshmind_gap_table.csv†L15-L17】
+
+## Pipeline Reliability and Maintenance Intelligence
+- Build a consolidation planner that reasons about ADD/UPDATE/DELETE operations, contradiction detection, and human-in-the-loop review, complementing existing dedupe heuristics.【F:research/meshmind_exceed_recommendations.md†L16-L20】【F:research/meshmind_gap_table.csv†L5-L7】
+- Expand maintenance automation with TTL enforcement, scope-level resets, replayable change logs, and backpressure-aware scheduling across Celery tasks and admin tooling.【F:research/meshmind_exceed_recommendations.md†L10-L23】【F:research/ai_memory_features_catalog.csv†L14-L31】
+- Offer change-data-capture hooks and event streams so downstream analytics or cache layers can react to graph updates in real time.【F:research/ai_memory_features_catalog.csv†L14-L59】
+
+## Retrieval Quality and Evaluation
+- Publish an open evaluation harness that benchmarks Recall@k, MRR, NDCG, latency, and token costs for each retrieval recipe across vector, hybrid, and graph-traversal scenarios.【F:research/meshmind_exceed_recommendations.md†L31-L33】【F:research/ai_memory_features_catalog.csv†L24-L27】
+- Provide curated datasets and synthetic corpora that stress-test consolidation, summarization, and scoping logic for reproducible validation runs.【F:research/meshmind_exceed_recommendations.md†L16-L23】【F:research/ai_memory_features_catalog.csv†L24-L29】
+- Integrate online feedback loops (implicit clicks, rerank overrides) to continuously tune scoring weights and LLM-assisted rerank prompts.【F:research/meshmind_exceed_recommendations.md†L7-L9】【F:research/ai_memory_features_catalog.csv†L11-L18】
+
+## Experience and Ecosystem Expansion
+- Release browser ingestion extensions and MCP-compatible capture workflows that preserve provenance metadata while streaming memories into MeshMind.【F:research/meshmind_exceed_recommendations.md†L32-L33】【F:research/ai_memory_features_catalog.csv†L25-L29】
+- Launch hosted and managed MeshMind tiers with tenancy isolation, billing telemetry, and migration paths from self-hosted installations.【F:research/meshmind_gap_table.csv†L10-L13】【F:research/ai_memory_features_catalog.csv†L16-L21】
+- Deliver turnkey starter kits (demo agents, notebooks, LangGraph templates) demonstrating best practices for each feature tier and vertical use case.【F:research/meshmind_exceed_recommendations.md†L19-L23】【F:research/ai_memory_features_catalog.csv†L21-L59】
+
+## Observability, Safety, and Operations
+- Instrument full OpenTelemetry traces, metrics, and structured logs across ingestion, maintenance, retrieval, and LLM calls with dashboards for latency, recall, and cost monitoring.【F:research/meshmind_exceed_recommendations.md†L16-L23】【F:research/meshmind_gap_table.csv†L12-L15】
+- Add safety guardrails: rate limiting, anomaly detection on ingestion payloads, abuse monitoring, and configurable moderation pipelines integrated with governance tooling.【F:research/meshmind_gap_table.csv†L10-L17】【F:research/ai_memory_features_catalog.csv†L20-L39】
+- Provide disaster-recovery playbooks including automated backups, time-travel restores (leveraging bi-temporal data), and chaos-testing scenarios for graph backends.【F:research/meshmind_exceed_recommendations.md†L4-L9】【F:research/meshmind_gap_table.csv†L2-L5】

ISSUES.md

@@ -17,6 +17,7 @@
 - [ ] Document Neo4j driver requirements and verify connectivity against a live cluster (CLI connectivity checks exist but still need validation against a real instance).
 - [ ] Exercise the new namespace/entity-label filtering against live Memgraph/Neo4j datasets to confirm Cypher predicates behave as expected.
 - [ ] Regenerate `uv.lock` to reflect the updated dependency set (`pymgclient`, `fastapi`, `uvicorn`, extras) so CI tooling stays in sync.
+- [x] Break down the competitive roadmap (MCP parity, multi-level scoping, bi-temporal edges, advanced rerankers, governance) into executable epics with owners and timelines referencing `GOALS.md`/`ROADMAP.md` (`ROADMAP_TASKS.md` now captures atomic work including MCP search parity, chat-store compatibility, crew scopes, summaries, and embedder routing).
 ## Medium Priority
 - [x] Persist results from consolidation and compression tasks back to the database (currently in-memory only).
 - [x] Refine `Memory.importance` scoring to reflect actual ranking heuristics instead of a constant.

PLAN.md

@@ -43,3 +43,9 @@
 2. **Operational Observability** – Export telemetry to Prometheus/OpenTelemetry and surface dashboards/alerts.
 3. **Celery Hardening** – Stress test consolidation/compression heuristics at scale and codify retry/backoff policies.
 4. **Model Fidelity** – Replace compatibility shims with production-ready Pydantic models once dependency support catches up.
+
+## Phase 6 – Competitive Roadmap Alignment (New)
+1. **Strategic Goals** – Track `GOALS.md` as the repository for parity, governance, and differentiation targets derived from the latest competitor analyses.
+2. **Planning Options** – Reference `PLANNING_THOUGHTS.md` during iteration reviews to select the sequencing strategy (parity-first, reliability-first, differentiator-led) that matches current constraints and staffing.
+3. **Execution Roadmap** – Groom tasks against `ROADMAP.md`, prioritizing MCP parity, multi-level scoping, bi-temporal edges, advanced rerankers, and governance items before lower-impact enhancements.
+4. **Roadmap Task Granularity** – Keep `ROADMAP_TASKS.md` synchronized with competitive research by covering MCP search/graph tools, chat-store persistence, crew-sharing scopes, conversation summaries, and embedder routing so execution tickets remain atomic.

PLANNING_THOUGHTS.md

@@ -0,0 +1,24 @@
+# Planning Thoughts
+
+## Option A – Parity-First Sprints
+- **Concept**: Deliver the competitor feature set (MCP server, bi-temporal graph, scoped tenancy, RRF/MMR rerankers) before layering differentiators so MeshMind can immediately replace Mem0, Graphiti, and Zep in pilot programs.【F:research/meshmind_exceed_recommendations.md†L4-L15】【F:research/meshmind_gap_table.csv†L2-L13】
+- **Why it works**: Rapidly closes critical capability gaps, simplifies messaging (“everything they have, plus more”), and unlocks co-marketing with ecosystem partners reliant on MCP integrations and hosted tenancy workflows.【F:research/ai_memory_features_catalog.csv†L16-L59】
+- **Risks**: Compresses bandwidth for experimentation, leaving differentiators (personalization, governance, evaluation harness) for later and risking burnout if parity demands exceed available engineering cycles.【F:research/meshmind_exceed_recommendations.md†L24-L33】
+- **Mitigations**: Parallelize observability and evaluation harness groundwork so readiness reviews keep quality high, and schedule design spikes for differentiator features while parity builds progress.【F:research/meshmind_exceed_recommendations.md†L16-L33】
+
+## Option B – Reliability and Observability First
+- **Concept**: Fortify graph durability, telemetry, tenancy, and governance before shipping parity features to ensure every new capability launches with enterprise-grade reliability and compliance hooks.【F:research/meshmind_exceed_recommendations.md†L9-L32】【F:research/meshmind_gap_table.csv†L3-L17】
+- **Why it works**: Positions MeshMind as the safest, most trustworthy platform, attracting regulated customers and enabling paid tiers/SLAs sooner than pure feature parity could.【F:research/meshmind_gap_table.csv†L10-L17】
+- **Risks**: Competitive demos may still highlight missing MCP tooling or advanced retrieval features, slowing adoption within open-source agent ecosystems that expect immediate compatibility.【F:research/meshmind_gap_table.csv†L6-L13】
+- **Mitigations**: Release public roadmap updates, partner with early adopters on co-developed MCP pilots, and provide interim adapters or compatibility layers until full parity arrives.【F:research/meshmind_exceed_recommendations.md†L4-L15】
+
+## Option C – Differentiator-Led Sequencing
+- **Concept**: Invest early in personalization, governance analytics, evaluation harnesses, and hosted offerings to leapfrog competitors while continuing incremental parity work in parallel tracks.【F:research/meshmind_exceed_recommendations.md†L24-L33】【F:research/meshmind_gap_table.csv†L11-L17】
+- **Why it works**: Creates a compelling “MeshMind advantage” narrative (adaptive retrieval, safety, hosted tier) that can justify premium pricing or open new verticals even if some parity items arrive later.【F:research/ai_memory_features_catalog.csv†L24-L59】
+- **Risks**: Without MCP parity or multi-level scoping, integrators might face friction onboarding, reducing the immediate utility of differentiation investments.【F:research/meshmind_gap_table.csv†L3-L13】
+- **Mitigations**: Define must-have parity milestones (MCP server beta, scoping primitives) as release gates for differentiator GA and staff shared teams to maintain progress across both tracks.【F:research/meshmind_exceed_recommendations.md†L4-L23】
+
+## Cross-Cutting Considerations
+- Maintain a living roadmap that sequences parity, reliability, and differentiation work with clear dependencies so contributors can volunteer for the highest-leverage streams.【F:research/meshmind_exceed_recommendations.md†L4-L33】
+- Prioritize documentation updates (SDK guides, governance policies, evaluation harness instructions) alongside feature work to keep MeshMind’s onboarding advantage intact.【F:research/ai_memory_features_catalog.csv†L21-L59】
+- Schedule recurring competitive reviews to ingest new Mem0/Zep/Graphiti releases and adjust priority ordering before each planning increment.【F:research/meshmind_gap_table.csv†L2-L17】

RECOMMENDATIONS.md

@@ -14,6 +14,8 @@
 - Introduce evaluation loops for the new importance heuristic (e.g., LLM-assisted ranking or analytics-driven weights) to tune thresholds over time, leveraging the telemetry stats now emitted.
 - Exercise the new `llm_client` overrides via REST/gRPC integration smoke tests (once credentials are available) to confirm per-request models/endpoints behave consistently outside unit tests.
 - Expand predicate/registry management APIs beyond the CLI helper so services can manage vocabularies programmatically.
+- Align roadmap execution artifacts with competitor features by scheduling MCP search/graph endpoints, chat-store persistence,
+  shared crew scopes, conversation summarization, and multi-embedder routing work from the refreshed `ROADMAP_TASKS.md` list.
 - Plan for reintroducing full Pydantic models once packaging support is aligned with target Python versions.
 
 ## Improve Developer Experience
@@ -31,6 +33,8 @@
   the Makefile and expand it as new developer utilities are introduced. Keep `SETUP.md` synchronized when dependencies change.
 - Provide walkthroughs for configuring LLM reranking, including sample prompts and response expectations.
 - Add onboarding notes for the REST/gRPC service layers with sample payloads and curl/grpcurl snippets.
+- Keep `GOALS.md`, `PLANNING_THOUGHTS.md`, and `ROADMAP.md` refreshed each planning cycle so contributors have a single
+  reference for competitive priorities, sequencing strategies, and upcoming feature commitments.
 
 ## Future Enhancements
 - Export telemetry to Prometheus/OpenTelemetry and wire alerts/dashboards around ingestion and maintenance.

RESUME_NOTES.md

@@ -22,6 +22,10 @@
 - Extended `DUMMIES.md` and `docs/testing.md` to capture the `FakeLLMClient` behaviour and the setup script smoke-test
   coverage; updated `ENVIRONMENT_NEEDS.md` and `NEEDED_FOR_TESTING.md` to acknowledge that optional packages now install with
   network access.
+- Authored competitor-aligned planning collateral (`GOALS.md`, `PLANNING_THOUGHTS.md`, `ROADMAP.md`) and refreshed planning
+  artifacts (`PLAN.md`, `SOT.md`, `RECOMMENDATIONS.md`, `TODO.md`, `ISSUES.md`) to surface roadmap-aligned follow-up work.
+- Expanded `ROADMAP_TASKS.md` to cover MCP search/graph parity, chat-store compatibility, crew/shared scopes, conversation
+  summaries, and multi-embedder routing; mirrored the updates across planning/backlog documents.
 
 ## Environment State
 
@@ -34,12 +38,14 @@
 
 ## Next Session Starting Points
 
-1. Work through the remaining `TODO.md` priority items that are unblocked by missing infrastructure (e.g., research tasks may
+1. Convert new `ROADMAP_TASKS.md` subtasks (MCP search parity, chat-store API, crew scopes, conversation summaries, embedder
+   routing) into detailed design docs and engineering tickets.
+2. Work through the remaining `TODO.md` priority items that are unblocked by missing infrastructure (e.g., research tasks may
    remain pending until live services exist).
-2. Validate Neo4j connectivity end-to-end once a reachable instance is available, using `meshmind admin graph --backend neo4j`.
-3. Plan integration tests for the LLM override payloads against a real provider when credentials are provisioned; update
+3. Validate Neo4j connectivity end-to-end once a reachable instance is available, using `meshmind admin graph --backend neo4j`.
+4. Plan integration tests for the LLM override payloads against a real provider when credentials are provisioned; update
    `docs/testing.md` accordingly.
-4. Continue chipping away at shim retirements documented in `DUMMIES.md`, starting with replacing the Pydantic compatibility
+5. Continue chipping away at shim retirements documented in `DUMMIES.md`, starting with replacing the Pydantic compatibility
    layer when production targets allow the real dependency.
 
 ## Helpful References