Add Librarian — AI chat panel for SQL lineage Q&A

[liliyaminibaeva]

PR: Add Librarian — AI-powered chat panel for SQL lineage Q&A

Summary

Librarian is a new chat panel that lets users ask natural-language questions about their data using SQL lineage context and uploaded PDF documentation. It sits as a third resizable panel to the right of the analysis view.

Features

• AI Chat with structured responses (Summary / Data Lineage / Documentation sections); off-topic questions get a fixed refusal.
• Multiple AI providers: OpenAI, Anthropic, and custom OpenAI-compatible endpoints (e.g. LiteLLM). Config in localStorage, sent only to the configured provider.
• PDF documentation — drag-and-drop or click upload. Local pipeline: pdfjs-dist text extraction → 500-char chunks → Xenova/multilingual-e5-small embedding (Web Worker, 100+ languages) → cosine similarity search. 10 MB per file, no file count limit.
• Inline identifier highlighting in answers — table and column names are styled distinct from inline code; matching is case-insensitive and normalized to canonical schema casing.
• Click-to-navigate: clicking an assistant answer reads its Summary section and drives the lineage view via the existing search pipeline. Every referenced column highlights across all owning tables.
• Schema search in the Schema view header — substring match by table or column name with Prev/Next cycling.
• Per-project state isolation (RAM-only) — chat history, PDFs, and embedded chunks are scoped to the active project; switching projects does not leak content into another project's prompt. F5 wipes everything.
• Keyboard shortcut ⌘L / Ctrl+L to toggle the panel.
• Test coverage — 304 Vitest tests passing (yarn typecheck clean).

How it works

User question → use-librarian-chat.ts (orchestrator)
  ├── Lineage context ← lineage-formatter.ts ← useLineageState()
  ├── SQL snippet ← useProject() (active file, truncated to 3000 chars)
  ├── PDF context ← vector search ← embedding-service (Web Worker)
  └── Chat history ← Zustand store (last 10 messages sent to AI, full history shown in UI)
        ↓
  context-builder.ts → assembled prompt
        ↓
  ai-service.ts → fetch() → OpenAI / Anthropic / Custom endpoint
        ↓
  Response → store → chat UI (markdown + highlighted identifiers)

• All processing runs locally in the browser (embeddings via Web Worker, no backend).
• AI answers strictly from provided data (SQL, lineage, PDFs) — no general knowledge.
• API keys stored in localStorage, sent only to the configured provider.

Files added

app/src/features/librarian/
├── components/          # UI: panel, chat, PDF upload, settings dialog, toggle
├── services/            # AI service, context builder, lineage formatter, PDF processor, vector search, embeddings
├── hooks/               # Chat orchestrator, sync-active-project
├── workers/             # Embedding Web Worker (Xenova/transformers)
├── utils/               # schema-identifiers (detect, resolve, extractSummary)
├── __tests__/           # Vitest suites
├── types.ts
├── constants.ts
├── store.ts
└── index.ts

app/public/polly-icon.svg                 # Custom duck icon
app/src/lib/lineage-node-resolver.ts      # Resolve ChatReference[] → lineage node ids
app/src/lib/lineage-navigation.ts         # Pure consumer of NavigationTarget for the lineage tab
app/src/lib/__tests__/lineage-node-resolver.test.ts
app/src/lib/__tests__/lineage-navigation.test.ts
docs/librarian.md                         # User guide

Files modified

File	Change
app/src/components/Workspace.tsx	Third ResizablePanel for Librarian; toggle button in analysis toolbar; LibrarianPanelWithNavigation wires chat-click → lineage search-term + reveal
app/src/components/AnalysisView.tsx	Wires actionsRef.current.revealNodeInGraph into applyLineageNavigation deps
app/src/components/GlobalDropZone.tsx	Overlay made pointer-events-none so PDF drops reach the Librarian dropzone
app/src/lib/view-state-store.ts	Added librarianOpen flag and per-project Librarian state hook integration
app/src/lib/shortcuts.ts	Toggle-librarian shortcut
app/src/lib/navigation-context.tsx	NavigationTarget.highlightNodeIds / tablesToExpand / primaryFocusId for chat-click navigation
app/package.json	Added pdfjs-dist, @xenova/transformers, Vitest + RTL deps
app/vitest.config.ts	New — Vitest configuration
CHANGELOG.md	Single [Unreleased] entry describing the Librarian feature
docs/librarian.md	User guide

New dependencies

Package	Purpose	Size
pdfjs-dist	PDF text extraction in browser	~800 KB
@xenova/transformers	Local embedding model inference	~2 MB JS + ~23 MB model (lazy loaded, cached in env.useBrowserCache)
vitest, @testing-library/react, jsdom	Testing (devDependencies)	—

Known limitations

• Single-substring search. Click-to-navigate writes one substring into the lineage search box; heterogeneous Summary references (e.g. both MANDT and BUKRS) cannot share a single search term — the first column wins. The user can dismiss the highlight via the lineage search box.
• SchemaView highlights one table at a time. The Schema view's selection prop accepts a single selectedTableName; schema search uses Prev/Next cycling to compensate.
• Chat-click overrides active manual lineage search. Setting the search term on click overwrites whatever the user had typed; clearing the search box resets to no highlight.
• Per-project state is RAM-only. Page reload (F5) clears chat, PDFs, and embedded chunks. By design — no localStorage persistence to keep the LLM context predictable.

Test plan

• Open FlowScope, paste SQL, verify lineage renders normally.
• Click the Librarian toolbar button (or ⌘L) — Librarian panel opens on the right.
• Configure AI settings (OpenAI, Anthropic, or custom endpoint).
• Ask a question about the SQL — verify structured response with Summary / Data Lineage / Documentation.
• Upload a PDF (click and drag-and-drop) — verify it processes and shows "ready" status.
• Ask a question related to PDF content — verify Documentation section cites the PDF.
• Click an assistant answer that mentions a column — verify the lineage view highlights the column across all owning tables and pans/pulses on a source table.
• Click an answer that mentions only a table — verify pan/pulse on that table.
• Switch projects — verify chat, PDFs, and answers are scoped to the active project (no leakage).
• Run yarn typecheck — 0 errors.
• Run yarn test — all tests pass.

[twoxfh]

Very cool! I tried it out and its very useful. From what I have seen, prompts are also part of the configuration in these instances. Could the context-builder source its prompt from an editable field with the default being the current prompt?

Context can get huge, user facing raw string size might be very helpful. Just a thought!