docs: cutting-edge search + memory research report for Flatbread + Proof#185
Conversation
… Proof
Synthesizes a /proof DAG run that executed in three ranks:
- Rank 1 (parallel research streams):
- Audit of Flatbread's current retrieval surface (filter DSL, refs, transformers).
- Audit of @flatbread/proof's parent->child context-passing pipeline.
- SOTA dossiers on dense/sparse/hybrid retrieval, structural/graph retrieval,
agent-memory architectures, and embeddable runtimes (sqlite-vec, Orama, etc.).
- Rank 2 (parallel synthesis):
- Proof-track opportunity catalog (O1..O11).
- Flatbread-track opportunity catalog (F1..F12).
- Novel/industry-shifting bets (N1..N12).
- Rank 3:
- Composes the final 13-section weighing report at repo root.
Recommends Posture B (Effort-Graph + Hybrid Retrieval), shows three phased
trajectories (V0 foundation / V1 minimum credible / V2 industry-shifting), and
proposes new packages (@flatbread/index, @flatbread/embed, @flatbread/mcp,
@flatbread/memory) plus an MCP surface that lets Proof use Flatbread as its
retrieval backend over the host repo.
Co-authored-by: Tony <tonyketcham@users.noreply.github.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Found two review-worthy issues in flatbread-search-and-memory-research.md.
- The report cites
/tmp/proof-search/*artifacts andfile:///tmp/...links that are not checked into the repo, so readers cannot inspect the evidence chain from GitHub or a fresh clone. - It presents
flatbread index verify/flatbread index buildas runnable CLI commands, but the currentflatbreadCLI only exposesstart,init, andcodegen.
Because this PR is docs-only, I did not identify any test coverage asks beyond keeping the citations and command surface accurate.
Sent by Cursor Automation: Flatbread PR Review
|
|
||
| ### 2.1 Frontmatter-only retrieval is structurally insufficient | ||
|
|
||
| The audit ([`/tmp/proof-search/audit-flatbread-retrieval.md`](file:///tmp/proof-search/audit-flatbread-retrieval.md) §2.4–2.5, §3.1, §5) is unambiguous about today's retrieval surface in [`packages/core`](packages/core): |
There was a problem hiding this comment.
This link target is local to the author's machine/runtime, not to the repo. None of the cited audit-*, sota-*, or synthesis-* markdown files are checked in anywhere under this tree, so a reader opening the PR or a fresh clone cannot inspect the source material behind the report's claims.
Please either commit those dossiers under a stable repo path and link them relatively, or rewrite these as plain-text citations to committed/external sources instead of file:///tmp/... links.
|
|
||
| #### F9 — Index artifacts + verify CLI | ||
|
|
||
| chunks/edges jsonl plus gitignored sqlite/orama binaries; `flatbread index verify` checks hashes **Pros:** CI drift detection; clone size control **Hypothesis:** Support load shifts from mystique to actionable rebuild commands. |
There was a problem hiding this comment.
flatbread index verify is written here as if it already exists, but the current CLI in packages/flatbread/src/cli/index.ts only defines start, init, and codegen. Copy/pasting this from the doc will fail today.
If this is future work, please mark it explicitly as proposed/not-yet-shipped instead of command voice. Otherwise, land the command and link the implementation/docs from this report.
|
|
||
| **Query surfaces:** GraphQL remains the introspectable contract for apps; MCP exposes tool-shaped facades over the identical resolver stack; codegen grows typed accessors (document nodes + thin SDK) so agents rarely hand-author GraphQL strings. | ||
|
|
||
| **Index format:** Source markdown stays canonical. **Tier A:** committed `chunks.jsonl` / `edges.jsonl` manifests for CI and small repos. **Tier B (default at scale):** derived `sqlite` with FTS5 + `sqlite-vec` (or Orama snapshot) under `.flatbread/index/`, gitignored, rebuilt by `flatbread index build` with manifest hashes recorded. Optional **Tier C:** team policy commits quantized vectors per N3 when audit demands byte-level review. |
There was a problem hiding this comment.
Same issue here for flatbread index build: the report treats it as an existing command, but there is no such subcommand in the current CLI. That makes the "rebuilt by flatbread index build" workflow unverifiable for readers.
Please either implement and document the subcommand before describing it as runnable, or rephrase this section as a proposed future design rather than present-day operational guidance.
…x CLI - Add docs/research/performant-context-search/README.md describing the Proof DAG methodology and why intermediate dossiers are not vendored; link to root report and packages/proof README (.flatbread/artifacts default). - Remove file:///tmp and /tmp/proof-search links from the research report; point readers at the methodology README, repo paths, and §13 references. - Mark flatbread index build/verify as proposed future CLI; cite shipped commands in packages/flatbread/src/cli/index.ts (start/init/codegen only). - Soften embeddable-runtime guidance to regenerated .flatbread/index outputs vs committing manifests where desired. Co-authored-by: Tony <tonyketcham@users.noreply.github.com>
tonyketcham
force-pushed
the
toeknee/search-research-dag-719a
branch
from
49018d1 to
af7e118
Compare
2 participants
Summary of changes
Strategic research report on performant retrieval / agent memory for Flatbread + Proof, plus clone-safe citations and honest CLI wording.
Layout (updated)
docs/research/performant-context-search/flatbread-search-and-memory-research.mdaudit-*,sota-*,synthesis-*):docs/research/performant-context-search/intermediate/— section-aligned excerpts duplicated from the merged report so evidence matches the original Proof DAG structure without/tmporfile://links.docs/research/performant-context-search/dags/performant-context-search.json(10 tasks / 3 ranks).docs/research/performant-context-search/README.mddocuments default transcripts under<cwd>/.flatbread/artifacts/…and optional--full-output-dir docs/research/performant-context-search/proof-artifacts/…. Seeproof-artifacts/README.md.flatbread-search-and-memory-research.mdpoints readers at the new location.Review fixes (prior commits)
file:///tmpcitations; methodology README + §13 references.flatbread index build/verifycalled proposed; cite shippedpackages/flatbread/src/cli/index.ts.pnpm exec prettier --writeon touched markdown/jsonDoes this introduce any non-backwards compatible changes?
Does this include any user config changes?