JS: Incremental API graph #20733
JS: Incremental API graph #20733
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
|
||
| pragma[nomagic] | ||
| private predicate shouldBacktrackIntoOverlay(DataFlow::SourceNode nd) { | ||
| exists(DataFlow::Node overlayNode | |
Check warning
Code scanning / CodeQL
Omittable 'exists' variable Warning
| predicate inScope(DataFlow::Node node) { any() } | ||
| } | ||
|
|
||
| private module Full = Stage<FullInput>; |
Check warning
Code scanning / CodeQL
Dead code Warning
|
|
||
| private module Full = Stage<FullInput>; | ||
|
|
||
| query predicate missingDefNode(DataFlow::Node node) { |
Check warning
Code scanning / CodeQL
Dead code Warning
| not exists(MkDef(node)) | ||
| } | ||
|
|
||
| query predicate missingUseNode(DataFlow::Node node) { |
Check warning
Code scanning / CodeQL
Dead code Warning
| not exists(MkUse(node)) | ||
| } | ||
|
|
||
| query predicate lostEdge(Node pred, Label::ApiLabel lbl, Node succ) { |
Check warning
Code scanning / CodeQL
Dead code Warning
| not Cached::edge(pred, lbl, succ) | ||
| } | ||
|
|
||
| query predicate counts(int numEdges, int numOverlayEdges, float ratio) { |
Check warning
Code scanning / CodeQL
Dead code Warning
asgerf
force-pushed
the
js/incremental-api-graphs
branch
from
2e5779a to
c8abbdc
Compare
asgerf
force-pushed
the
js/incremental-api-graphs
branch
from
c8abbdc to
e1a113c
Compare
asgerf
force-pushed
the
js/incremental-api-graphs
branch
2 times, most recently
from
3cb97e6 to
fbc775f
Compare
asgerf
force-pushed
the
js/incremental-api-graphs
branch
4 times, most recently
from
bde7253 to
69857e3
Compare
Simply wraps everything in 'cached private module Stage {}' and adds 'import Stage'.
The diff is large because of indentation changes.
This is needed for localizing ApiLabel later
This was initially lost after rebasing with indentation changes
Some abstract classes defines fields without binding them, leaving it up to the subclasses to bind them. When combined with overlay[local?], the charpred for such an abstract class can become local, while the subclasses are global. The means the charpred needs to be materialized, even though it doesn't bind the fields, leading to a cartesian product.
This was somehow lost in a rebase
We want the type itself to be local but nearly all its member predicates are global.
Previously this was implied by MkClassInstance but that's no longer the case.
asgerf
force-pushed
the
js/incremental-api-graphs
branch
from
69857e3 to
5fbf877
Compare
There was a problem hiding this comment.
Pull request overview
This PR implements incremental API graph analysis for JavaScript through a two-stage approach. In the base database, stage 1 performs the full analysis while stage 2 does nothing. In overlay databases, stage 1 does nothing while stage 2 only analyzes changed files and their dependencies, using stage 1 results as roots. This optimization can reduce end-to-end analysis time by up to 40% in some cases.
- Introduces a staged architecture with configurable roots and scope for each stage
- Refactors API graph node construction to be more selective based on stage configuration
- Updates overlay annotations across multiple framework files to support incremental analysis
Reviewed changes
Copilot reviewed 32 out of 32 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
javascript/ql/lib/semmle/javascript/ApiGraphs.qll |
Core implementation of two-stage incremental API graph analysis with new stage-based architecture, tracking predicates, and debug utilities |
javascript/ql/lib/semmle/javascript/frameworks/Templating.qll |
Changes concat to strictconcat and updates overlay annotation from global to local? for IncludeFunctionAsEntryPoint |
javascript/ql/lib/semmle/javascript/frameworks/*.qll |
Adds overlay[local?] annotations to various EntryPoint classes across multiple framework files |
javascript/ql/lib/semmle/javascript/ES2015Modules.qll |
Adds pragma[nomagic] annotations and refactors reExportsAs to improve join ordering |
javascript/ql/lib/semmle/javascript/security/dataflow/*.qll |
Adds overlay[global] annotations to abstract sink and flow label classes |
python/ql/lib/semmle/python/frameworks/data/internal/ApiGraphModels.qll |
Adds overlay[local?] annotations to TypeModelUseEntry and TypeModelDefEntry classes |
ruby/ql/lib/codeql/ruby/frameworks/data/internal/ApiGraphModels.qll |
Adds overlay[local?] annotations to TypeModelUseEntry and TypeModelDefEntry classes |
javascript/ql/test/library-tests/frameworks/Redux/test.expected |
Updates expected test output to reflect shorter, more readable API graph node descriptions |
javascript/ql/test/library-tests/frameworks/Knex/test.expected |
Updates expected test output with simplified node descriptions |
javascript/ql/test/library-tests/EndpointNaming/EndpointNaming.expected |
Updates expected test output with more concise node descriptions |
javascript/ql/test/ApiGraphs/custom-entry-point/VerifyAssertions.ql |
Adds overlay[local?] annotation to CustomEntryPoint class |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
1 participant
Makes API graphs incremental.
The high-level overview is that we run the underlying data flow analysis in two global stages, where each stage is configured with a set of "roots" and a set of nodes that are "in scope". They're configured in a way where the first stage only does real work in the base, and the second stage only does any real work in the overlay.
Based on the tracking results from the first stage, we identify which nodes could flow into the overlay part, and use those as roots in the second stage.
Commit-by-commit review strongly recommended.
See backilnked performance evaluations. The performance gains are very swingy; in many cases it doesn't make much of a difference, but then in some cases e2e time drops by 40%.