feat: move Vanir signature generation to a cron job#5126
Merged
cuixq merged 33 commits intogoogle:masterfrom Apr 2, 2026
Merged
feat: move Vanir signature generation to a cron job#5126cuixq merged 33 commits intogoogle:masterfrom
cuixq merged 33 commits intogoogle:masterfrom
Conversation
Contributor
Author
|
/gemini review |
Contributor
Author
|
/gemini review |
Contributor
Author
|
/gemini review |
Member
michaelkedar
left a comment
There was a problem hiding this comment.
This looks mostly good to me, I'm just a bit concerned on how gracefully errors are handled in general
michaelkedar
previously approved these changes
Apr 1, 2026
another-rex
reviewed
Apr 2, 2026
Contributor
another-rex
left a comment
There was a problem hiding this comment.
From our offline discussions:
- Add a vanir generation modified date so that we can track what version of this vuln record the vanir generation is for (so we know to skip it when regening)
- For the initial merge, let's just skip all records that already have vanir signatures, because it's going to take far too long to do. Reenable regening once we have the vanir git optimisations
another-rex
reviewed
Apr 2, 2026
Contributor
another-rex
left a comment
There was a problem hiding this comment.
Just looked at the last change, which LGTM. Just 1 question.
michaelkedar
previously approved these changes
Apr 2, 2026
michaelkedar
approved these changes
Apr 2, 2026
another-rex
approved these changes
Apr 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#4611
This PR introduces a new
vanir_signaturesworker to enrich OSV records with Vanir signatures. This cron job identifies modified vulnerabilities, generates the signatures, and updates both the Datastore and GCS with the enriched data.ThreadPoolExecutorto handle parallel I/O for fetching records from GCS and managing batch processing.--dry-runflag for testing the workflow without modifying production data.modify_storage_client_adaptersto configure the GCS client with an increasedHTTPAdapterconnection pool and retries.Currently the crob job is scheduled daily at 9:00 AM Sydney time which we may want to adjust to run more frequent later.