Add continuous graph updates via Git webhook and poll watcher#615
Open
Add continuous graph updates via Git webhook and poll watcher#615
Conversation
- Add api/git_utils/incremental_update.py with incremental_update(), fetch_remote(), get_remote_head(), and repo_local_path() helpers - Export new functions from api/git_utils/__init__.py - Add POST /api/webhook endpoint with HMAC-SHA256 validation, branch filtering, and repo URL matching - Add background poll watcher via FastAPI lifespan (_poll_loop, _poll_all_repos, _poll_repo) - Add WEBHOOK_SECRET, TRACKED_BRANCH, POLL_INTERVAL env vars - Document new env vars in .env.template - Add tests/test_webhook.py with unit tests" Co-authored-by: gkorland <753206+gkorland@users.noreply.github.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Copilot
AI
changed the title
[WIP] Add continuous graph updates via Git webhook
Add continuous graph updates via Git webhook and poll watcher
Mar 13, 2026
Resolve PR #615 conflicts, preserve the continuous update configuration, and stabilize the webhook incremental-update tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
|
@CodeRabbit review |
Contributor
✅ Actions performedReview triggered.
|
Contributor
|
Important Review skippedBot user detected. To trigger a single review, invoke the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughThis pull request implements continuous incremental graph updates via webhooks and background polling. It adds configuration for webhook secrets, branch tracking, and polling intervals; introduces the incremental update engine for diff-based graph synchronization; exposes a POST /api/webhook endpoint with HMAC-SHA256 signature validation; and includes a background poll-watcher task with comprehensive test coverage. Changes
Sequence Diagram(s)sequenceDiagram
actor Client as Push Event/<br/>Polling Loop
participant API as FastAPI Server<br/>/api/webhook
participant Repo as Git Repository
participant Analyzer as Source Analyzer
participant Graph as FalkorDB<br/>Graph Database
participant Redis as Redis<br/>(Bookmark)
alt Webhook Path
Client->>API: POST /api/webhook<br/>(with signature & payload)
API->>API: Validate HMAC-SHA256<br/>Verify branch match
API->>Repo: Extract repo from payload
else Polling Path
Client->>API: Background poll-watcher<br/>triggers periodically
API->>Repo: Fetch remote & check HEAD
end
API->>Repo: Resolve from_sha, to_sha<br/>(current & latest)
Repo->>API: Return commit objects
API->>Repo: Compute file diff<br/>(added, modified, deleted)
Repo->>API: Return file changeset
API->>Repo: Checkout target commit
Repo->>Repo: Update working tree
Note over Graph,Analyzer: Process Changed Files
par Remove Deleted/Modified
API->>Graph: DELETE nodes & edges<br/>for removed/modified files
and Analyze & Insert Changed
API->>Analyzer: Analyze added/<br/>modified files
Analyzer->>API: Return AST/symbols
API->>Graph: INSERT new nodes/<br/>UPDATE existing edges
end
Graph->>Graph: Return change summary<br/>(added, modified, deleted counts)
API->>Redis: Persist new commit SHA<br/>as bookmark
Redis->>API: Acknowledge
API->>Client: Return 200 with<br/>update summary
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
📝 Coding Plan
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a continuous/incremental graph update mechanism to avoid full re-indexes on every repo change, integrating both a push-webhook trigger and a background poll-watcher into the FastAPI backend.
Changes:
- Added
api/git_utils/incremental_update.pyto compute file-level diffs between two SHAs and update the FalkorDB graph incrementally while persisting the new commit bookmark in Redis. - Added
POST /api/webhookplus URL-matching helpers and FastAPI lifespan-managed poll-watcher to trigger incremental updates automatically. - Documented new environment variables and the new webhook endpoint in
.env.templateandREADME.md, plus added unit tests for the webhook and incremental-update helpers.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
api/index.py |
Adds webhook endpoint, URL matching, poll-watcher loop, and lifespan wiring to trigger incremental updates. |
api/git_utils/incremental_update.py |
Implements diff-based incremental update flow (checkout, delete stale file nodes, re-analyze changed files, update Redis bookmark). |
api/git_utils/__init__.py |
Replaces wildcard export with explicit exports, including the new incremental update helpers. |
tests/test_webhook.py |
Adds unit tests for URL matching, webhook behavior (open/secured), and basic incremental_update edge cases. |
README.md |
Documents new env vars and the /api/webhook endpoint. |
.env.template |
Adds WEBHOOK_SECRET, TRACKED_BRANCH, and POLL_INTERVAL configuration. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@api/git_utils/incremental_update.py`:
- Around line 172-180: The current update deletes definitions for
deleted+modified files via g.delete_files and re-analyzes only added+modified
via analyzer.analyze_files, which leaves inbound callers/importers stale; after
computing files_to_add and files_to_remove, collect the transitive set of
dependent files that import or call those changed files (using the graph g's
reverse edges / dependency lookup), add those dependent filenames to
files_to_add (excluding already-deleted files), and then call
analyzer.analyze_files on this expanded files_to_add so
SourceAnalyzer.analyze_files reprocesses untouched callers and restores inbound
edges; ensure you use g's dependency/query methods (the graph instance g) to
find dependents before calling analyzer.analyze_files.
- Around line 166-185: Wrap the entire mutation sequence that performs
repo.checkout_tree/to_commit set_head_detached, Graph(repo_name) updates,
analyzer.analyze_files(...) and set_repo_commit(...) in an exclusive repo-scoped
lock (keyed on repo_name) so concurrent runs for the same repo cannot
interleave; acquire the lock before calling repo.checkout_tree and hold it until
after set_repo_commit (or until commit of all graph changes), use a
distributed-lock primitive if you have multiple workers, set a sensible timeout
and ensure the lock is always released in a finally/cleanup block and that
errors are logged/propagated while still releasing the lock.
In `@api/index.py`:
- Around line 497-505: The code calls incremental_update(repo_name, before,
after) using the webhook payload.before; instead, retrieve the stored bookmark
via get_repo_commit(repo_name) and use that as from_sha when calling
incremental_update (i.e., pass get_repo_commit(repo_name) as the first/from
argument), but detect mismatch: if the stored bookmark is missing or does not
equal the graph's current commit that lines up with the push history (or cannot
reach payload.before), fall back to performing a full reindex of the repo (call
the existing full reindex routine) rather than running a partial
incremental_update; update the _update closure (and its call site around
repo_local_path, fetch_remote, and loop.run_in_executor) to implement this
branching logic so the graph never advances past gaps.
- Around line 114-122: The webhook endpoint is anonymously writable because
WEBHOOK_SECRET defaults to empty; update startup and the /api/webhook handler so
webhook auth is mandatory: at startup (when reading WEBHOOK_SECRET) fail fast
with a clear error if it's empty in production mode, or modify the webhook route
handler to apply token_required when WEBHOOK_SECRET is not set (use the existing
token_required decorator) so the mutating endpoint is never unauthenticated;
ensure you update the webhook handler (the /api/webhook function) to prefer HMAC
verification when WEBHOOK_SECRET is set and fallback to token_required
otherwise, and add a clear log message indicating which auth mode is in effect.
- Around line 455-470: The webhook handler currently enforces GitHub-specific
headers and payload fields (it reads X-Hub-Signature-256 into sig_header, builds
expected_sig from WEBHOOK_SECRET, and reads repository.clone_url into repo_url),
which rejects GitLab webhooks; update the logic in the handler to detect GitLab
deliveries by checking for X-Gitlab-Token or X-Gitlab-Signature when
X-Hub-Signature-256 is absent, validate the token/signature using the configured
secret (respecting GitLab’s verification method), and when parsing payload fall
back to repository.git_http_url or project.git_http_url if repository.clone_url
is missing; alternatively, if you prefer to keep GitHub-only behavior, update
documentation to state the webhook supports GitHub only and explicitly fail with
a clear message when GitLab headers are present.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: cea312f7-19cf-400b-a510-c28520dca5e1
📒 Files selected for processing (6)
.env.templateREADME.mdapi/git_utils/__init__.pyapi/git_utils/incremental_update.pyapi/index.pytests/test_webhook.py
Reprocess dependent files during incremental updates, add repo-scoped update locking, harden webhook auth and provider handling, and fall back to full reindex when the stored bookmark no longer matches incoming history. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Sanitize error response in webhook endpoint to avoid exposing internal exception details (information exposure via str(exc)) - Normalize SHA format in incremental_update no-op response to use consistent short SHA (to_sha[:7]) matching the non-noop path - Add comprehensive poll-watcher unit tests covering: missing clone, fetch failure, up-to-date skip, behind trigger, no remote head, and missing bookmark scenarios Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…nalysis When _full_reindex_repository is triggered (e.g., after a force-push or missing bookmark), the working tree may still be at the old commit. The reindex would analyze stale files and set the bookmark to the old commit, creating an infinite retry loop. Now accepts target_sha and checks out the target commit before analysis when a local clone exists. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
gkorland
requested changes
Mar 18, 2026
Contributor
gkorland
left a comment
gkorland
left a comment
There was a problem hiding this comment.
Good feature addition overall — the incremental update engine, webhook endpoint, and poll-watcher are well-structured. A few issues to address before merging.
- api/graph.py: Add _VALID_ENTITY_LABELS allowlist to get_entity_at_position
to prevent Cypher injection via f-string label interpolation
- api/graph.py: Extract _make_falkordb_connection() helper; delete_graph_if_exists
now accepts optional db param so callers can share a single connection
- api/index.py: Move pygit2 imports (CheckoutStrategy, Repository) to module-level;
add subprocess and re to module-level imports
- api/index.py: Replace fragile prefix-based SHA comparison with git rev-parse
to resolve stored short SHA to full 40-char form for unambiguous comparison;
validate SHA hex format before passing to git
- api/index.py: Use raise ... from exc for webhook JSON parse error
- api/analyzers/{python,java,csharp}/analyzer.py: Fix graph param type to Graph
and return types to list[Entity | ResolvedEntityRef] in resolve_type,
resolve_method, resolve_symbol; add ResolvedEntityRef and Graph imports
- tests/test_webhook.py: Update poll-watcher tests to mock subprocess.run;
add command assertions in mock callbacks
Co-authored-by: gkorland <753206+gkorland@users.noreply.github.com>
Contributor
Code Review SummaryGood feature overall — incremental update engine, webhook endpoint, and poll-watcher are well-structured with solid test coverage. Six issues flagged as inline comments:
See inline comments for suggested fixes. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Full re-indexing on every change is too slow for large repos. This adds an incremental update engine that computes a diff between two commit SHAs and only touches affected files, plus two trigger modes: a GitHub/GitLab push webhook and a background poll watcher.
Core engine —
api/git_utils/incremental_update.pyincremental_update(repo_name, from_sha, to_sha, ignore=[])— resolves both SHAs via pygit2, classifies file changes (added/modified/deleted), checks outto_sha, deletes stale nodes/edges, re-analyses changed files, and persists the new commit bookmark in Redis. Idempotent (from_sha == to_shais a no-op). Accepts abbreviated or full SHAs.fetch_remote(repo_path)—git fetch originvia subprocessget_remote_head(repo_path, branch)— returns remote branch HEAD SHArepo_local_path(repo_name)— resolves clone path; respectsREPOSITORIES_DIRenv overrideWebhook endpoint —
POST /api/webhookAccepts GitHub/GitLab push event payloads. When
WEBHOOK_SECRETis set, validatesX-Hub-Signature-256withhmac.compare_digest(timing-safe). Ignores pushes to untracked branches (200 response, no retry). Resolves the target repo by URL-matching against indexed repos (normalises.gitsuffix and case).Background poll watcher
Started via FastAPI lifespan on startup (cancelled cleanly on shutdown). At each
POLL_INTERVALtick, fetches all indexed repos, compares stored commit SHA againstorigin/<TRACKED_BRANCH>, and callsincremental_updateif behind. Handles short vs. full SHA comparison correctly (prefix match only when lengths differ).Configuration — new env vars (documented in
.env.template)WEBHOOK_SECRETTRACKED_BRANCHmainPOLL_INTERVAL600disables the watcherOriginal prompt
💬 Send tasks to Copilot coding agent from Slack and Teams to turn conversations into code. Copilot posts an update in your thread when it's finished.
Summary by CodeRabbit
Release Notes
New Features
Documentation