Add compatibility fallbacks and maintenance persistence

.github/workflows/ci.yml

@@ -0,0 +1,44 @@
+name: CI
+
+on:
+  push:
+    branches: ["main", "review", "review-1"]
+  pull_request:
+
+env:
+  PIP_DISABLE_PIP_VERSION_CHECK: "1"
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check out
+        uses: actions/checkout@v4
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+      - name: Install toolchain
+        run: |
+          pip install uv
+          uv pip install -e .
+          uv pip install ruff pyright typeguard toml-sort yamllint
+      - name: Lint and format checks
+        run: make fmt-check
+
+  tests:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check out
+        uses: actions/checkout@v4
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+      - name: Install dependencies
+        run: |
+          pip install uv
+          uv pip install -e .
+          uv pip install pytest
+      - name: Run pytest
+        run: make test

AGENTS.md

@@ -0,0 +1,11 @@
+# Agent Instructions
+
+## Documentation Workflow
+- After each batch of changes, add a `CHANGELOG.md` entry with an ISO 8601 date/time stamp and developer-facing detail (files, modules, functions, variables, and rationale). Every commit should correspond to a fresh entry.
+- Maintain `README.md` as the canonical description of the project; update it whenever behaviour or workflows change. Archive older versions separately when requested.
+- After each iteration, refresh `ISSUES.md`, `SOT.md`, `PLAN.md`, `RECOMMENDATIONS.md`, `TODO.md`, and related documentation to stay in sync with the codebase.
+
+## Style Guidelines
+- Use descriptive Markdown headings starting at level 1 for top-level documents.
+- Keep lines to 120 characters or fewer when practical.
+- Prefer bullet lists for enumerations instead of inline commas.

CHANGELOG.md

@@ -0,0 +1,67 @@
+# Changelog
+
+## [2025-02-15T00:45:00Z]
+### Added
+- Introduced `meshmind/retrieval/graph.py` with hybrid/vector/regex/exact/BM25/fuzzy wrappers that hydrate candidates from the active `GraphDriver` before delegating to existing scorers, plus `meshmind/tests/test_graph_retrieval.py` to verify namespace filtering and hybrid integration.
+- Added `meshmind/cli/admin.py` and wired `meshmind/cli/__main__.py` to expose `admin` subcommands for predicate management, maintenance telemetry, and graph connectivity checks; created `meshmind/tests/test_cli_admin.py` to cover the new flows.
+- Created `meshmind/tests/test_neo4j_driver.py` and a `Neo4jGraphDriver.verify_connectivity` helper to exercise driver-level sanity checks without a live cluster.
+- Logged importance score distributions via `meshmind/pipeline/preprocess.summarize_importance` so telemetry captures mean/stddev/recency metrics after scoring.
+
+### Changed
+- Updated `MeshMind` search helpers (`meshmind/client.py`) to auto-load memories from the configured driver when `memories` is `None`, reusing the new graph-backed wrappers.
+- Reworked `meshmind/pipeline/consolidate.py` to return a `ConsolidationPlan` with batch/backoff thresholds and skipped-group tracking; `meshmind/tasks/scheduled.consolidate_task` now emits skip counts and returns a structured summary.
+- Tuned Python compatibility metadata to `>=3.11,<3.13` in `pyproject.toml` and refreshed docs (`README.md`, `NEEDED_FOR_TESTING.md`, `SOT.md`) accordingly.
+- Enhanced `meshmind/pipeline/preprocess.py` to emit telemetry gauges for importance scoring and added `meshmind/tests/test_pipeline_preprocess_store.py::test_score_importance_records_metrics`.
+- Expanded retrieval, CLI, and driver test coverage (`meshmind/tests/test_retrieval.py`, `meshmind/tests/test_tasks_scheduled.py`) to account for graph-backed defaults and new return types.
+
+### Documentation
+- Updated `README.md`, `PROJECT.md`, `PLAN.md`, `SOT.md`, `FINDINGS.md`, `DISCREPANCIES.md`, `RECOMMENDATIONS.md`, `NEEDED_FOR_TESTING.md`, `ISSUES.md`, and `TODO.md` to describe graph-backed retrieval wrappers, CLI admin tooling, consolidation backoff behaviour, telemetry metrics, and revised Python support.
+- Copied the refreshed README guidance into `README_OLD.md` as an archival reference while keeping `README.md` as the primary source.
+
+## [2025-10-14T14:57:47Z]
+### Added
+- Introduced `meshmind/_compat/pydantic.py` to emulate `BaseModel`, `Field`, and `ValidationError` when Pydantic is unavailable, enabling tests to run in constrained environments.
+- Added `meshmind/testing/fakes.py` with `FakeMemgraphDriver`, `FakeRedisBroker`, and `FakeEmbeddingEncoder`, plus a package export and dedicated pytest coverage (`meshmind/tests/test_db_drivers.py`, `meshmind/tests/test_tasks_scheduled.py`).
+- Created heuristics-focused test cases for consolidation outcomes, maintenance tasks, and the revised retrieval dispatcher to guarantee behaviour without external services.
+
+### Changed
+- Replaced the constant importance assignment in `meshmind/pipeline/preprocess.score_importance` with a heuristic that factors token diversity, recency, metadata richness, and embedding magnitude.
+- Rebuilt `meshmind/pipeline/consolidate` around a `ConsolidationOutcome` dataclass that merges metadata, averages embeddings, and surfaces removal IDs; `meshmind/tasks/scheduled.consolidate_task` now applies updates and deletes duplicates lazily via `_get_manager`/`_reset_manager` helpers.
+- Hardened Celery maintenance tasks by logging driver initialization failures, tracking update counts, and returning deterministic totals; compression counts now reflect the number of persisted updates.
+- Updated `meshmind/core/similarity`, `meshmind/retrieval/bm25`, and `meshmind/retrieval/fuzzy` with pure-Python fallbacks so numpy, scikit-learn, and rapidfuzz remain optional.
+- Adjusted `meshmind/pipeline/extract.extract_memories` to defer `openai` imports until a default client is required, unblocking DummyLLM-driven tests.
+- Reworked `meshmind/retrieval/search.search` to rerank the original (filtered) candidate ordering, prepend reranked results, and append hybrid-sorted fallbacks, preventing index drift when rerankers return relative positions.
+- Normalised SQLite entity hydration in `meshmind/db/sqlite_driver._row_to_dict` so JSON metadata is decoded only when stored as strings.
+- Refreshed pytest fixtures (`meshmind/tests/conftest.py`, `meshmind/tests/test_pipeline_preprocess_store.py`) to use deterministic encoders and driver doubles, ensuring CRUD and retrieval suites run without live services.
+
+### Documentation
+- Promoted `README.md` as the single source of truth (archiving the previous copy in `README_OLD.md`) and documented the new heuristics, compatibility shims, and test doubles.
+- Updated `NEEDED_FOR_TESTING.md` with notes about the compatibility layer, optional dependencies, and fake drivers.
+- Reconciled `PROJECT.md`, `ISSUES.md`, `PLAN.md`, `SOT.md`, `RECOMMENDATIONS.md`, `DISCREPANCIES.md`, `FINDINGS.md`, `TODO.md`, and `CHANGELOG.md` to capture the new persistence behaviour, heuristics, fallbacks, and remaining roadmap items.
+
+## [Unreleased] - 2025-02-14
+### Added
+- Configurable graph driver factory with in-memory, SQLite, Memgraph, and optional Neo4j implementations plus supporting tests.
+- REST and gRPC service layers (with FastAPI stub fallback) for ingestion and retrieval, including coverage in the test suite.
+- Observability utilities that collect metrics and structured logs across pipelines and scheduled Celery tasks.
+- Docker Compose definition provisioning Memgraph, Redis, and a Celery worker for local development.
+- Vector-only, regex, exact-match, and optional LLM rerank retrieval helpers with reranker utilities and exports.
+- MeshMind client wrappers for hybrid, vector, regex, and exact searches plus driver accessors.
+- Example script demonstrating triplet storage and diverse retrieval flows.
+- Pytest fixtures for encoder and memory factories alongside new retrieval tests that avoid external services.
+- Makefile targets for linting, formatting, type checks, and tests, plus a GitHub Actions workflow running lint and pytest.
+- README_LATEST.md capturing the current implementation and CHANGELOG.md for release notes.
+
+### Changed
+- Settings now surface `GRAPH_BACKEND`, Neo4j, and SQLite options while README/NEEDED_FOR_TESTING document the expanded setup.
+- README, README_LATEST, and NEW_README were consolidated so the promoted README reflects current behaviour.
+- PROJECT, PLAN, SOT, FINDINGS, DISCREPANCIES, ISSUES, RECOMMENDATIONS, and TODO were refreshed to capture new capabilities and
+  re-homed backlog items under a "Later" section.
+- Updated `SearchConfig` to support rerank models and refreshed MeshMind documentation across PROJECT, PLAN, SOT, FINDINGS,
+  DISCREPANCIES, RECOMMENDATIONS, ISSUES, TODO, and NEEDED_FOR_TESTING files.
+- Revised `meshmind.retrieval.search` to apply filters centrally, expose new search helpers, and integrate reranking.
+- Exposed graph driver access on MeshMind and refreshed retrieval-facing examples and docs.
+
+### Fixed
+- Example ingestion script now uses MeshMind APIs correctly and illustrates relationship persistence.
+- Tests rely on fixtures rather than deprecated hooks, improving portability across environments without Memgraph/OpenAI.

DISCREPANCIES.md

@@ -0,0 +1,53 @@
+# README vs Implementation Discrepancies
+
+## Overview
+- The legacy README has been superseded by `README.md`, which now reflects the implemented feature set.
+- The current codebase delivers extraction, preprocessing, triplet persistence, CRUD helpers, and expanded retrieval strategies
+  that were missing when the README was written.
+- Remaining gaps primarily involve pushing retrieval workloads into the graph backend, exporting observability to external sinks, and automated infrastructure provisioning.
+
+## API Surface
+- ✅ `MeshMind` now exposes CRUD helpers (`create_memory`, `update_memory`, `delete_memory`, `list_memories`, triplet helpers)
+  that the README referenced implicitly.
+- ✅ Triplet storage routes through `store_triplets` and `MemoryManager.add_triplet`, calling `GraphDriver.upsert_edge`.
+- ⚠️ The README still references `register_entity`, `register_allowed_predicates`, and `add_predicate`; predicate management is
+  handled automatically but there is no public API matching those method names.
+- ⚠️ README snippets showing `mesh_mind.store_memory(memory)` should be updated to call `store_memories([memory])` or the new
+  CRUD helpers.
+
+## Retrieval Capabilities
+- ✅ Vector-only, regex, exact-match, hybrid, BM25, fuzzy, and optional LLM rerank searches exist in `meshmind.retrieval.search`
+  and are surfaced through `MeshMind` helpers.
+- ⚠️ README implies retrieval queries the graph directly. Search helpers now fetch candidates from the configured driver when no
+  list is supplied but still score results in Python; Memgraph/Neo4j-native search remains future work.
+- ⚠️ Named helpers like `search_facts` or `search_procedures` never existed; the README should reference the dispatcher plus
+  specialized helpers now available.
+
+## Data & Relationship Modeling
+- ✅ Predicates are persisted automatically when storing triplets and tracked in `PredicateRegistry`.
+- ⚠️ README examples that look up subjects/objects by name still do not match the implementation, which expects UUIDs. Add
+  documentation explaining how to resolve names to UUIDs before storing edges.
+- ⚠️ Consolidation and expiry run via Celery jobs; README narratives should highlight that heuristics require further validation even though persistence is now wired up.
+
+## Configuration & Dependencies
+- ✅ `README.md` and `NEEDED_FOR_TESTING.md` document required environment variables, dependency guards, and setup steps.
+- ⚠️ README still omits optional tooling now required by the Makefile/CI (ruff, pyright, typeguard, toml-sort, yamllint);
+  highlight these prerequisites more prominently.
+- ✅ Python version support in `pyproject.toml` now pins `>=3.11,<3.13`, matching the dependency landscape documented in the README.
+
+## Example Code Paths
+- ✅ Updated example scripts demonstrate extraction, triplet creation, and multiple retrieval strategies.
+- ⚠️ Legacy README code that instantiates custom Pydantic entities remains inaccurate; extraction returns `Memory` objects and
+  validates `entity_label` names only.
+- ⚠️ Search examples should be updated to show the new helper functions and optional rerank usage instead of nonexistent
+  `search_facts`/`search_procedures` calls.
+
+## Tooling & Operations
+- ✅ Makefile and CI workflows now exist, aligning with README promises about automation once the README is refreshed.
+- ✅ Docker Compose now provisions Memgraph, Redis, and a Celery worker; README sections should highlight the workflow and
+  caveats for environments lacking container tooling.
+- ⚠️ Celery tasks still depend on optional infrastructure; README should clarify that heuristics and scheduling need production hardening even though persistence now works.
+
+## Documentation State
+- Continue promoting `README.md` as the authoritative guide and propagate updates to supporting docs
+  (`SOT.md`, `PLAN.md`, `NEEDED_FOR_TESTING.md`).

FINDINGS.md

@@ -0,0 +1,36 @@
+# Findings
+
+## General Observations
+- Core modules are now wired through the `MeshMind` client, including CRUD, triplet storage, and retrieval helpers. Graph-backed wrappers fetch candidates from the configured driver automatically; remaining integration work focuses on server-side query optimisation and heuristic evaluation loops.
+- Optional dependencies are largely guarded behind lazy imports, compatibility shims, or factory functions, improving portability. Environments still need to install tooling referenced by the Makefile and CI (ruff, pyright, typeguard, toml-sort, yamllint).
+- Documentation artifacts (`README.md`, `SOT.md`, `NEEDED_FOR_TESTING.md`) stay current when updated with each iteration; the legacy README has been archived as `README_OLD.md`.
+
+## Dependency & Environment Notes
+- `MeshMind` defers driver creation until persistence is required, enabling workflows without `pymgclient` and selecting between in-memory, SQLite, Memgraph, or Neo4j backends. CLI helpers (`meshmind admin graph`) now expose connectivity sanity checks.
+- Project metadata now advertises Python `>=3.11,<3.13`, aligning with available wheels for optional dependencies.
+- Encoder registration occurs during bootstrap, but custom deployments must ensure compatible models are registered before
+  extraction or hybrid search.
+- The OpenAI embedding adapter still expects dictionary-like responses; adapting to SDK objects remains on the backlog.
+- Celery tasks initialize lazily, yet Redis/Memgraph services are still required at runtime. Docker Compose now provisions both
+  services alongside a worker for local testing.
+
+## Data Flow & Persistence
+- Triplet storage now persists relationships and tracks predicates automatically, closing an earlier data-loss gap.
+- Consolidation and compression utilities now persist updates through the maintenance tasks, enforce batch/backoff thresholds, and surface skipped groups; larger-scale validation remains necessary.
+- Importance scoring uses heuristics (token diversity, recency, metadata richness, embedding magnitude) and now records telemetry summaries; continued evaluation will raise retrieval quality.
+
+## CLI & Tooling
+- CLI ingestion bootstraps encoders and entities automatically and now ships `admin` subcommands for predicate maintenance, telemetry dumps, and graph connectivity checks. External backends still require valid credentials and running services.
+- The Makefile introduces lint, format, type-check, and test targets, plus Docker helpers. External tooling installation is
+  required before targets succeed.
+- GitHub Actions now run formatting checks and pytest on push/PR, providing basic CI coverage.
+
+## Testing & Quality
+- Pytest suites rely on fixtures (`memory_factory`, `dummy_encoder`) and compatibility shims to run without external services. Coverage now includes graph-backed retrieval wrappers, Neo4j connectivity shims, CLI admin flows, and Celery consolidation telemetry.
+- Type checking via `pyright` and runtime checks via `typeguard` are exposed in the Makefile; dependency installation is
+  necessary for full validation.
+
+## Documentation
+- `README.md`/`README_LATEST.md` document setup, pipelines, retrieval, tooling, and now highlight service interfaces and observability.
+- Supporting docs (`ISSUES.md`, `PLAN.md`, `RECOMMENDATIONS.md`, `SOT.md`) reflect the latest capabilities and highlight remaining
+  gaps, aiding onboarding and future planning.

ISSUES.md

@@ -0,0 +1,34 @@
+# Issues Checklist
+
+## Blockers
+- [x] MeshMind client fails without `mgclient`; introduce lazy driver initialization or documented in-memory fallback.
+- [x] Register a default embedding encoder (OpenAI or sentence-transformers) during startup so extraction and hybrid search can run.
+- [x] Update OpenAI integration to match the current SDK (Responses API payload, embeddings API response structure).
+- [x] Replace eager `tiktoken` imports in `meshmind.core.utils` and `meshmind.pipeline.compress` with guarded, optional imports.
+- [x] Align declared Python requirement with supported dependencies (project now pins Python >=3.11,<3.13).
+
+## High Priority
+- [x] Implement relationship persistence (`GraphDriver.upsert_edge`) within the storage pipeline and expose triplet APIs.
+- [x] Restore high-level API methods promised in README (`register_entity`, predicate management, `add_memory`, `update_memory`, `delete_memory`).
+- [x] Ensure CLI ingestion registers entity models and embedding encoders or fails fast with actionable messaging.
+- [x] Provide configuration documentation and examples for Memgraph, Redis, and OpenAI environment variables.
+- [x] Add automated tests or smoke checks that run without external services (mock OpenAI, stub Memgraph driver).
+- [x] Create real docker-compose services for Memgraph and Redis or remove the placeholder file.
+- [ ] Document Neo4j driver requirements and verify connectivity against a live cluster (CLI connectivity checks exist but still need validation against a real instance).
+
+## Medium Priority
+- [x] Persist results from consolidation and compression tasks back to the database (currently in-memory only).
+- [x] Refine `Memory.importance` scoring to reflect actual ranking heuristics instead of a constant.
+- [x] Add vector, regex, and exact-match search helpers to match stated feature set or update documentation to demote them.
+- [x] Harden Celery tasks to initialize dependencies lazily and log failures when the driver is unavailable.
+- [ ] Validate consolidation heuristics on larger datasets and add conflict-resolution strategy when merged metadata conflicts.
+- [ ] Revisit the compatibility shim once production environments support Pydantic 2.x so the real models can be restored.
+- [ ] Push graph-backed retrieval into Memgraph/Neo4j search capabilities once available (current wrappers still score results in Python).
+- [ ] Reconcile tests that depend on `Memory.pre_init` and outdated OpenAI interfaces with the current implementation.
+- [x] Add linting, formatting, and type-checking tooling to improve code quality.
+
+## Low Priority / Nice to Have
+- [x] Offer alternative storage backends (in-memory driver, SQLite, etc.) for easier local development.
+- [x] Provide an administrative dashboard or CLI commands for listing namespaces, counts, and maintenance statistics (CLI admin subcommands now expose predicates, telemetry, and graph checks).
+- [ ] Publish onboarding guides and troubleshooting FAQs for contributors.
+- [ ] Explore plugin registration for embeddings and retrieval strategies to reduce manual wiring.