fix: prevent silent hang in --delta sync when Local API ignores edited.since#85
Open
lavarius wants to merge 1 commit into
Open
fix: prevent silent hang in --delta sync when Local API ignores edited.since#85lavarius wants to merge 1 commit into
lavarius wants to merge 1 commit into
Conversation
…es edited.since The Tana Desktop Local API silently ignores the `edited.since` filter on `/nodes/search` (verified via direct probe: six different `since` values from 5 minutes through epoch all return identical paginated results). On a 739K-node workspace this turns delta-sync into a 1h+ hang as it pages through the entire graph as if everything changed. This is an upstream Tana Desktop bug — `supertag-cli` sends the filter correctly; the server discards it. Until that's fixed, mitigate from the client: - Per-page progress logging (page 1, then every 10th page) so the hang stops being silent. - Auto-scaled abort cap: 25% of `sync_metadata.total_nodes`, with a 1000-page fallback if total_nodes is missing. The broken-API failure mode returns ~100% of the workspace, so any sub-100% threshold trips it; 25% leaves plenty of headroom for legitimately large deltas while capping wasted work. Override via `DeltaSyncOptions.maxPages`. - On abort, `sync_metadata.last_sync_timestamp` is NOT advanced — so the next delta-sync replays from the same point and re-merges the same rows idempotently. Error message points users to `supertag sync index` for a full resync. - Wire `logger` through `commands/sync.ts` to `DeltaSyncService` so the new progress lines actually surface in the CLI (previously the service defaulted to a no-op logger when invoked from the command layer). Tests: 7 new pagination tests covering progress logging, auto-scaling from total_nodes, explicit override, and that last_sync_timestamp is preserved on abort. Test fixture seed bumped from total_nodes=1000 to 1000000 so the new auto-scaled cap doesn't trip unrelated pagination tests. All 19 tests in delta-sync-pagination green. Verified live on the 739K-node workspace: cap fired at 1849 pages (25% of total / 100), aborted cleanly with the documented error.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR
supertag sync index --deltasilently hangs for an hour or more on large workspaces. Root cause: the Tana Desktop Local API drops theedited.sincefilter on/nodes/search, so what's supposed to be an incremental sync walks the entire graph. This PR doesn't fix the upstream filter (that's on Tana). It makes the failure visible and bounded so users don't wait an hour to find out something's wrong, and it points them to the fast recovery path (supertag sync index, no--delta).This is a safety patch. The delta-sync feature itself is only as good as the upstream filter; until that's fixed, full-refresh is the practical workflow.
Symptom
On a 739K-node workspace:
No progress, no error, eventually completes with most of the workspace re-merged. First-time users will assume the CLI is broken or hung.
Root cause (upstream, not this CLI)
Direct probe of the Tana Desktop Local API, bypassing this CLI entirely:
Six orders of magnitude of
sincevalues, all return identical paginated results. The filter is being silently dropped. This is a Tana Desktop bug, not asupertag-clibug. To my knowledge, this is the first surfaced report (no open or closed issues on this repo mentionedited.since,/nodes/search, or delta-sync pagination behavior).The wider context: full sync is much faster anyway
While debugging this, I measured all three sync transports end-to-end on the same 739K-node workspace:
supertag-export run mainsupertag sync indexsupertag sync index --delta(working)supertag sync index --delta(broken, today)So a full refresh is
~13 seconds totalon a 739K-node graph. Even if the upstream filter is fixed tomorrow,--deltaonly wins when the actual changeset is small enough that page-by-page HTTP roundtrips beat a 9-second cached-snapshot download. That's a narrower regime than the current docs imply.This PR is not advocating to remove
--delta(there are legitimate uses: long-running watch loops, environments where Playwright is unavailable). But the recovery hint in the error message points users at the path they probably wanted anyway.Changes
src/services/delta-sync.tsceil(sync_metadata.total_nodes * 0.25 / 100)pages. The broken-API failure mode returns ~100% of the workspace, so any sub-100% threshold catches it; 25% leaves headroom for legitimately large deltas while bounding wasted work. Falls back to 1000 pages iftotal_nodesis missing.last_sync_timestampon abort. When the cap fires,sync_metadata.last_sync_timestampis not advanced. The next attempt replays from the same point and re-merges any rows that were already committed, idempotently. Error message tells the user to runsupertag sync index(no--delta) for full resync.src/types/local-api.tsmaxPagestoDeltaSyncOptionsso callers can override the auto-scaled cap.src/commands/sync.tsloggerthrough toDeltaSyncServicein both the standalonerunDeltaSyncpath and thewatchpath. Without this, the new progress lines defaulted to a no-op logger when invoked from the CLI and never surfaced. Drive-by but necessary for the fix to be visible to users.tests/unit/delta-sync-pagination.test.ts7 new tests:
total_nodes(24K nodes gives a 60-page cap).maxPagesoverride wins over the auto-scaled value.supertag sync indexhint.last_sync_timestampis unchanged; merged rows are present (idempotent replay).Fixture seed bumped:
total_nodes1000 to 1000000 so the new auto-scaled cap doesn't trip unrelated pagination tests.All 19 tests in
delta-sync-pagination.test.tspass. No other test files modified.Verification
Live-tested against the 739K-node workspace that originally exhibited the hang:
ceil(739455 * 0.25 / 100)), aborted with the documented error.Recommended follow-ups (out of scope for this PR)
--deltadocs to mention the full-sync alternative and its current speed advantage. (Not making doc changes in this PR to keep the scope tight.)--deltain favor of cached-snapshot refresh as the default sync path.Notes for reviewers
MAX_PAGES_RATIO = 0.25constant is the main tunable. I considered 0.5 (more headroom) but settled on 0.25 because the broken-API failure mode is so distinctive (whole-workspace return) that there's no risk of false-positive aborts at 25%, and lower thresholds reduce wasted work.last_sync_timestampsteady on abort is the subtle correctness property. It's what makes the abort safe to retry rather than corrupt-on-retry. Test 7 pins it.