Skip to content

fix(goals): stop the generic memory table from re-surfacing phantom goals in the VFS #225

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
khustup2 merged 2 commits into from
Jun 2, 2026
Merged

fix(goals): stop the generic memory table from re-surfacing phantom goals in the VFS #225

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
86e9b99
fix(goals): stop generic memory table from re-surfacing phantom goals…
khustup2 Jun 2, 2026
de8b2af
test(vfs): bring deeplake-fs coverage to ≥90% + raise resume-brief ti…
khustup2 Jun 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 13 additions & 4 deletions src/notifications/sources/resume-brief.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,10 +57,19 @@ const LOOKBACK = 5;
const SCAN_LIMIT = 40;

/** Hard cap on the lookup. DeeplakeApi.query retries ~3.5s on an unreachable
* endpoint; the SessionStart hook budget is 5s and fetchOrgStats already
* spends up to 1.5s before us. Race it so a slow backend degrades to a
* plain welcome instead of stalling the hook. */
const QUERY_TIMEOUT_MS = 1_500;
* endpoint; the SessionStart hook budget is 5s and fetchOrgStats is served
* from cache before us. Race it so an *unreachable* backend degrades to a
* plain welcome instead of stalling the hook.
*
* Sized to the real cold-backend latency, NOT an optimistic guess. Measured
* 2026-06-02: the first session against a cold backend takes ~1.9s for this
* query (warm: ~0.3s). The earlier 1.5s cap sat *below* that cold latency, so
* every fresh-open silently lost the race — withTimeout does NOT cancel the
* in-flight query, it just discarded the rows we were ~0.4s from receiving,
* and pickResumeBrief reported "no prior summary" with the data sitting right
* there. We already pay the cold latency; 3s lets us keep the payoff instead
* of throwing it away. Still well inside the 5s hook budget. */
const QUERY_TIMEOUT_MS = 3_000;

function withTimeout<T>(p: Promise<T>, ms: number, fallback: T): Promise<T> {
return new Promise<T>((resolve) => {
Expand Down
14 changes: 14 additions & 0 deletions src/shell/deeplake-fs.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -224,6 +224,20 @@ export class DeeplakeFs implements IFileSystem {
const rows = await client.query(sql);
for (const row of rows) {
const p = row["path"] as string;
// Goal/KPI-shaped paths belong exclusively to the dedicated
// hivemind_goals / hivemind_kpis tables. Pre-routing hook
// versions (<=0.7.4) wrote goals to the generic memory table
// as plain files; surfacing those here re-injects phantom
// goals into the VFS goal namespace — visible in `ls /goal/...`
// but absent from `hivemind goal list` (the CLI reads only the
// structured table). Skip them when the dedicated table is
// configured so the two views stay in sync. (When it is not
// configured, goal routing is off and these rows are the only
// copy, so we keep them.)
const kind = classifyPath(p);
if ((kind === "goal" && fs.goalsTable) || (kind === "kpi" && fs.kpisTable)) {
continue;
}
Comment on lines +237 to +240
Copy link
Copy Markdown
Collaborator

@efenocchi efenocchi Jun 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@CodeRabbit what does it do?

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Jun 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@efenocchi, this code change fixes a "phantom goal" bug in the VFS bootstrap. Here's what it does:

Background: The VFS (DeeplakeFs) has two storage layers for goals:

  1. A generic memory table (old approach) — older hook versions (≤0.7.4) wrote goal entries here as plain files.
  2. A dedicated hivemind_goals table (new approach) — the structured source of truth that the CLI (hivemind goal list) reads from.

The bug: When DeeplakeFs.create() bootstrapped the generic memory table, it loaded all rows indiscriminately — including any goal-shaped paths (e.g. /goal/<owner>/opened/<uuid>.md) that old hook versions had written there. This caused those goals to reappear in the VFS (ls /goal/... showed them) even though the structured table had no record of them → "phantom" goals.

The fix (the added lines):

const kind = classifyPath(p);
if ((kind === "goal" && fs.goalsTable) || (kind === "kpi" && fs.kpisTable)) {
  continue;
}

During bootstrap, for each row fetched from the generic memory table:

  • It classifies the path using classifyPath() to check if it looks like a goal or KPI path.
  • If it does and the corresponding dedicated table (goalsTable / kpisTable) is configured, it skips loading that row into the VFS cache.
  • The goal/KPI data is then sourced exclusively from the dedicated structured tables (bootstrapped separately in goalsBootstrap / kpisBootstrap).
  • If no dedicated table is configured (legacy/routing-off mode), the memory rows are kept as-is — they're the only copy.

This ensures the VFS and the CLI always show the same set of goals with no phantom duplicates.

fs.files.set(p, null);
fs.meta.set(p, {
size: Number(row["size_bytes"] ?? 0),
Expand Down
Loading
Loading