Document LLM overrides and refresh provisioning workflows

.github/workflows/ci.yml

@@ -0,0 +1,48 @@
+name: CI
+
+on:
+  push:
+    branches: ["main", "review", "review-1"]
+  pull_request:
+
+env:
+  PIP_DISABLE_PIP_VERSION_CHECK: "1"
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check out
+        uses: actions/checkout@v4
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+      - name: Install toolchain
+        run: |
+          pip install uv
+          uv pip install --system -e .
+          uv pip install --system ruff pyright typeguard toml-sort yamllint
+      - name: Lint and format checks
+        run: make fmt-check
+      - name: Docs guard
+        env:
+          BASE_REF: ${{ github.event.pull_request.base.sha || 'HEAD~1' }}
+        run: make docs-guard
+
+  tests:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check out
+        uses: actions/checkout@v4
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+      - name: Install dependencies
+        run: |
+          pip install uv
+          uv pip install --system -e .
+          uv pip install --system pytest
+      - name: Run pytest
+        run: make test

AGENTS.md

@@ -0,0 +1,19 @@
+# Agent Instructions
+
+## Documentation Workflow
+- After each batch of changes, add a `CHANGELOG.md` entry with an ISO 8601 date/time stamp in United States Eastern time (include the timezone code, e.g., `America/New_York` or `ET`) and developer-facing detail (files, modules, functions, variables, and rationale). Every commit should correspond to a fresh entry.
+- Maintain `README.md` as the canonical description of the project; update it whenever behaviour or workflows change. Archive older versions separately when requested.
+- Keep the `docs/` wiki and provisioning guides (`SETUP.md`, `ENVIRONMENT_NEEDS.md`) in sync with code updates; add or revise the
+  relevant page whenever features, modules, or workflows change.
+- After each iteration, refresh `ISSUES.md`, `SOT.md`, `PLAN.md`, `RECOMMENDATIONS.md`, `TODO.md`, and related documentation to stay in sync with the codebase.
+- Ensure `TODO.md` retains the `Completed`, `Priority Tasks`, and `Recommended Waiting for Approval Tasks` sections, moving finished items under `Completed` at the end of every turn.
+- Make every task in `TODO.md` atomic: each entry must describe a single, self-contained deliverable with enough detail to execute and verify without cross-referencing additional context.
+- Update `RESUME_NOTES.md` at the end of every turn so the next session starts with accurate context.
+- When beginning a turn, review `README.md`, `PROJECT.md`, `PLAN.md`, `RECOMMENDATIONS.md`, `ISSUES.md`, and `SOT.md` to harvest new actionable work. Maintain at least ten quantifiable, prioritised items in the `Priority Tasks` section of `TODO.md`, adding context or links when needed.
+- After completing any task, immediately update `TODO.md`, check for the next actionable item, and continue iterating until all unblocked `Priority Tasks` are exhausted for the session.
+- Continuously loop through planning and execution: finish a task, document it, surface new follow-ups, and resume implementation so long as environment blockers allow. If extra guidance would improve throughput, extend these instructions proactively.
+
+## Style Guidelines
+- Use descriptive Markdown headings starting at level 1 for top-level documents.
+- Keep lines to 120 characters or fewer when practical.
+- Prefer bullet lists for enumerations instead of inline commas.

CLEANUP.md

@@ -0,0 +1,88 @@
+# Cleanup Plan for Temporary Workarounds
+
+The following actions should be executed once the development environment consistently provides internet access and all
+optional dependencies can be installed. Each item references the affected files, the temporary workaround currently in
+place, and the exact remediation steps required to bring the implementation in line with production expectations.
+
+## Compatibility Layers
+
+### Replace Pydantic Shim
+- **Files**: `meshmind/_compat/pydantic.py`, modules importing from `meshmind._compat.pydantic`.
+- **Current State**: Custom `BaseModel` drop-in used when `pydantic` is unavailable.
+- **Action**:
+  1. Reintroduce `pydantic` as a hard dependency and update `pyproject.toml` extras accordingly.
+  2. Migrate all models to inherit from the official `pydantic` classes.
+  3. Delete `meshmind/_compat/pydantic.py` once no modules import it directly.
+  4. Update tests to cover validation using real `pydantic` features (e.g., `.model_dump`).
+
+### Retire FastAPI Stub
+- **Files**: `meshmind/api/rest.py`, `meshmind/api/service.py`, `docs/api.md`, `SETUP.md`.
+- **Current State**: A lightweight FastAPI replacement exposes REST endpoints without requiring the framework.
+- **Action**:
+  1. Install `fastapi`, `uvicorn`, and related extras via the setup scripts.
+  2. Replace the stub implementation with a true FastAPI app using pydantic request/response models.
+  3. Update tests to spin up the FastAPI test client instead of the stub.
+  4. Refresh documentation to reflect the production stack only.
+
+### Replace gRPC Dataclass Shim
+- **Files**: `meshmind/api/grpc.py`, `meshmind/tests/test_service_interfaces.py`, `docs/api.md`.
+- **Current State**: Dataclass-based stubs mirror generated proto classes.
+- **Action**:
+  1. Author protobuf definitions for service contracts and commit generated Python code.
+  2. Integrate `grpcio` and `grpcio-tools` into the setup scripts and lockfile.
+  3. Update the service module to use generated classes and channel/server implementations.
+  4. Convert tests to rely on `grpc.aio` test utilities and remove dataclass shims.
+
+## Task Scheduling Workarounds
+
+### Remove Celery Dummy App and Beat Fallback
+- **Files**: `meshmind/tasks/celery_app.py`, `meshmind/tasks/scheduled.py`.
+- **Current State**: Custom placeholders allow imports without Celery and Celery Beat.
+- **Action**:
+  1. Make `celery` a required dependency for maintenance tasks.
+  2. Refactor scheduling utilities to import real Celery constructs and fail fast when misconfigured.
+  3. Add integration tests that execute Celery workers against Redis or RabbitMQ in docker-compose.
+  4. Delete fallback classes once coverage exists.
+
+## Testing Fakes
+
+### Evaluate Graph/Embedding/Test Fakes
+- **Files**: `meshmind/testing/fakes.py`, `meshmind/tests/conftest.py`, `meshmind/tests/test_*`.
+- **Current State**: In-memory drivers and dummy encoders enable offline unit tests.
+- **Action**:
+  1. Keep unit-test fakes for isolation but ensure parallel integration suites use live services.
+  2. Document expectations in `docs/testing.md` and mark fakes clearly as test-only utilities.
+  3. Remove any production code paths that accidentally import fakes.
+
+### Replace Memgraph Client Mock
+- **Files**: `meshmind/tests/test_memgraph_driver.py`.
+- **Current State**: Monkeypatches a fake `mgclient` module for driver tests.
+- **Action**:
+  1. Install `pymgclient` and run tests against a dockerized Memgraph instance.
+  2. Retain the fake only for negative/offline coverage; guard it behind an explicit fixture flag.
+
+## Observability and Telemetry
+
+### Confirm Logging Metrics Alignment
+- **Files**: `meshmind/core/observability.py`, `meshmind/pipeline/*`, `docs/telemetry.md`.
+- **Current State**: Logging/metrics rely on standard library fallbacks.
+- **Action**:
+  1. Integrate preferred observability stack (e.g., OpenTelemetry) once dependencies are greenlit.
+  2. Remove or demote placeholders that mimic external exporters.
+
+## Tooling and Automation
+
+### Finalize Setup Scripts
+- **Files**: `run/install_setup.sh`, `run/maintenance_setup.sh`.
+- **Current State**: Scripts install packages opportunistically.
+- **Action**:
+  1. Review installed package list after environment unblock and remove conditional guards.
+  2. Ensure scripts configure CLI tools (e.g., `grpcurl`, `neo4j-admin`) that were previously skipped offline.
+
+### Lock Dependency Graph
+- **Files**: `pyproject.toml`, `uv.lock`.
+- **Current State**: Lockfile may exclude previously unavailable packages.
+- **Action**:
+  1. Regenerate `uv.lock` after all dependencies are installed.
+  2. Commit the updated lockfile and note differences in `CHANGELOG.md`.
+

DISCREPANCIES.md

@@ -0,0 +1,53 @@
+# README vs Implementation Discrepancies
+
+## Overview
+- The legacy README has been superseded by `README.md`, which now reflects the implemented feature set.
+- The current codebase delivers extraction, preprocessing, triplet persistence, CRUD helpers, and expanded retrieval strategies
+  that were missing when the README was written.
+- Remaining gaps primarily involve pushing retrieval workloads into the graph backend, exporting observability to external sinks, and automated infrastructure provisioning.
+
+## API Surface
+- ✅ `MeshMind` now exposes CRUD helpers (`create_memory`, `update_memory`, `delete_memory`, `list_memories`, triplet helpers)
+  that the README referenced implicitly.
+- ✅ Triplet storage routes through `store_triplets` and `MemoryManager.add_triplet`, calling `GraphDriver.upsert_edge`.
+- ⚠️ The README still references `register_entity`, `register_allowed_predicates`, and `add_predicate`; predicate management is
+  handled automatically but there is no public API matching those method names.
+- ⚠️ README snippets showing `mesh_mind.store_memory(memory)` should be updated to call `store_memories([memory])` or the new
+  CRUD helpers.
+
+## Retrieval Capabilities
+- ✅ Vector-only, regex, exact-match, hybrid, BM25, fuzzy, and optional LLM rerank searches exist in `meshmind.retrieval.search`
+  and are surfaced through `MeshMind` helpers.
+- ⚠️ README implies retrieval queries the graph directly. Search helpers now fetch candidates from the configured driver when no
+  list is supplied but still score results in Python; Memgraph/Neo4j-native search remains future work.
+- ⚠️ Named helpers like `search_facts` or `search_procedures` never existed; the README should reference the dispatcher plus
+  specialized helpers now available.
+
+## Data & Relationship Modeling
+- ✅ Predicates are persisted automatically when storing triplets and tracked in `PredicateRegistry`.
+- ⚠️ README examples that look up subjects/objects by name still do not match the implementation, which expects UUIDs. Add
+  documentation explaining how to resolve names to UUIDs before storing edges.
+- ⚠️ Consolidation and expiry run via Celery jobs; README narratives should highlight that heuristics require further validation even though persistence is now wired up.
+
+## Configuration & Dependencies
+- ✅ `README.md` and `ENVIRONMENT_NEEDS.md` document required environment variables, dependency guards, and setup steps.
+- ⚠️ README still omits optional tooling now required by the Makefile/CI (ruff, pyright, typeguard, toml-sort, yamllint);
+  highlight these prerequisites more prominently.
+- ✅ Python version support in `pyproject.toml` now pins `>=3.11,<3.13`, matching the dependency landscape documented in the README.
+
+## Example Code Paths
+- ✅ Updated example scripts demonstrate extraction, triplet creation, and multiple retrieval strategies.
+- ⚠️ Legacy README code that instantiates custom Pydantic entities remains inaccurate; extraction returns `Memory` objects and
+  validates `entity_label` names only.
+- ⚠️ Search examples should be updated to show the new helper functions and optional rerank usage instead of nonexistent
+  `search_facts`/`search_procedures` calls.
+
+## Tooling & Operations
+- ✅ Makefile and CI workflows now exist, aligning with README promises about automation once the README is refreshed.
+- ✅ Docker Compose now provisions Memgraph, Redis, and a Celery worker; README sections should highlight the workflow and
+  caveats for environments lacking container tooling.
+- ⚠️ Celery tasks still depend on optional infrastructure; README should clarify that heuristics and scheduling need production hardening even though persistence now works.
+
+## Documentation State
+- Continue promoting `README.md` as the authoritative guide and propagate updates to supporting docs
+  (`SOT.md`, `PLAN.md`, `ENVIRONMENT_NEEDS.md`, `docs/`).

DUMMIES.md

@@ -0,0 +1,17 @@
+# Dummies and Compatibility Artifacts
+
+The following table tracks all temporary shims, fake implementations, and stub services that were introduced while the
+sandbox lacked internet access or external infrastructure. Each row notes what the component does today and the
+recommended next step now that full dependencies can be installed.
+
+| Component | Location | Purpose | Current Usage | Recommended Action |
+| --- | --- | --- | --- | --- |
+| Pydantic compatibility shim | `meshmind/_compat/pydantic.py` | Provides a minimal `BaseModel` replacement when `pydantic` is unavailable. | Imported in service payloads and tests whenever the real package is missing. | Replace with real `pydantic` models once package installation is guaranteed; remove shim after migration. |
+| FastAPI REST stub | `meshmind/api/rest.py` (`RestAPIStub`, `create_app`) | Exposes REST behaviour without the FastAPI dependency. | Tests use the stub when FastAPI is not installed; production should use FastAPI. | Keep temporarily for offline tests but plan to gate it behind an explicit test flag once FastAPI becomes mandatory. |
+| gRPC dataclass shim | `meshmind/api/grpc.py` (`GrpcServiceStub` and request/response dataclasses) | Mimics generated gRPC classes so tests can exercise the service layer without `grpcio`. | Entire gRPC surface currently relies on this shim. | Replace with proto definitions and generated stubs when `grpcio-tools` is available; keep shim only for unit tests. |
+| Celery dummy app | `meshmind/tasks/celery_app.py` (`_DummyCeleryApp`) | Allows module imports when Celery is missing. | Unit tests rely on the dummy to avoid Celery; production should use real Celery. | Retire dummy once Celery is a hard dependency; until then ensure tests explicitly exercise the real app when installed. |
+| Celery beat fallback | `meshmind/tasks/scheduled.py` (`crontab` shim) | Supplies a no-op crontab when Celery beat is missing. | Prevents import errors in test environments without Celery. | Remove once Celery is required; otherwise guard usage with feature flags. |
+| Fake graph/storage drivers | `meshmind/testing/fakes.py` (`FakeMemgraphDriver`, `FakeRedisBroker`, `FakeEmbeddingEncoder`) | Provide offline stand-ins for Memgraph, Redis, and embedding models. | Pytest fixtures and documentation rely on these for isolation. | Keep as long as offline tests are desired; supplement with integration suites that use real services. |
+| Fake LLM client | `meshmind/testing/fakes.py` (`FakeLLMClient`) | Records per-request overrides and emits deterministic responses so tests exercise reranking without installing the OpenAI SDK. | Service/interface tests (`meshmind/tests/test_service_interfaces.py`, `test_client.py`) and the CLI fixtures inject this stub when `openai` is unavailable. | Keep for unit tests; add integration tests with real providers once keys and network access are provisioned. |
+| Dummy encoder fixture | `meshmind/tests/conftest.py` (`dummy_encoder`) and dependent tests | Supplies a lightweight embedding encoder for search tests. | Used across retrieval and service tests to avoid network calls. | Keep for unit tests; add integration coverage with real encoders once APIs are configured. |
+| Fake mgclient module | `meshmind/tests/test_memgraph_driver.py` (monkeypatch of `mgclient`) | Simulates the Memgraph client so driver code runs without the native binary. | Enables driver unit tests without installing `mgclient`. | Replace with real `mgclient`-backed tests once package access is ensured; keep shim for fallback coverage. |

Dockerfile

@@ -0,0 +1,24 @@
+FROM python:3.11-slim
+
+ENV PIP_NO_CACHE_DIR=1 \
+    PYTHONDONTWRITEBYTECODE=1 \
+    PYTHONUNBUFFERED=1
+
+WORKDIR /app
+
+RUN apt-get update \
+    && apt-get install -y --no-install-recommends \
+        build-essential \
+        cmake \
+        libssl-dev \
+        libkrb5-dev \
+        curl \
+        git \
+    && rm -rf /var/lib/apt/lists/*
+
+COPY . /app
+
+RUN pip install uv \
+    && uv pip install --system -e .[dev,docs,testing]
+
+CMD ["bash"]

ENVIRONMENT_NEEDS.md

@@ -0,0 +1,30 @@
+# Tasks for Human Project Manager
+
+- Keep the Python package layer aligned with the project extras during base image
+  refreshes. The `run/install_setup.sh` and `run/maintenance_setup.sh` scripts now
+  install the full optional stack (neo4j driver, `pymgclient`, Redis, Celery extras,
+  FastAPI/Uvicorn, LLM tooling, and developer linters/testers). Ensure cached
+  environments either run the maintenance script or bake these dependencies into the
+  image so cold starts do not regress coverage.
+- Provide system-level build dependencies for the graph drivers (e.g., `build-essential`,
+  `cmake`, `libssl-dev`, `libkrb5-dev`) so `pymgclient` (and its `mgclient` module) install cleanly.
+- Provision external services and credentials (compose files now exist under the project
+  root and `meshmind/tests/docker/`):
+  - Neo4j instance reachable from the execution environment with `NEO4J_URI`,
+    `NEO4J_USERNAME`, `NEO4J_PASSWORD` (defaults to `neo4j` / `meshminD123`).
+  - Memgraph instance with `MEMGRAPH_URI`, `MEMGRAPH_USERNAME`, `MEMGRAPH_PASSWORD`
+    (anonymous access acceptable locally).
+  - Redis instance with `REDIS_URL`.
+  - LLM provider API key (`LLM_API_KEY` or fallback `OPENAI_API_KEY`) plus any
+    alternative base URLs/models required for OpenRouter, Azure, or Google-hosted
+    endpoints so the new `llm_client` overrides can be exercised end-to-end.
+- Supply datasets/fixtures (future request) representing large knowledge graphs to
+  stress-test consolidation heuristics and pagination under load.
+- Maintain outbound package download access to PyPI and vendor repositories; this
+  session confirmed package installation works when the network is open, and future
+  sessions need the same capability to refresh locks or install new optional
+  integrations.
+- Enable Docker or container runtime access (future request) so the provided
+  `docker-compose.yml` files can run inside this environment; alternatively, provision
+  remote services accessible to CI.
+- Document credential management procedures and rotation cadence so secrets stay current.