Skip to content

feat: move Vanir signature generation to a cron job#5126

Merged
cuixq merged 33 commits intogoogle:masterfrom
cuixq:vanir
Apr 2, 2026
Merged

feat: move Vanir signature generation to a cron job#5126
cuixq merged 33 commits intogoogle:masterfrom
cuixq:vanir

Conversation

@cuixq
Copy link
Copy Markdown
Contributor

@cuixq cuixq commented Mar 25, 2026

#4611

This PR introduces a new vanir_signatures worker to enrich OSV records with Vanir signatures. This cron job identifies modified vulnerabilities, generates the signatures, and updates both the Datastore and GCS with the enriched data.

  • Identifies vulnerabilities modified since the last successful run by querying the Vulnerability Datastore index, ensuring efficient updates.
  • Utilizes the vanir library to perform batch processing of OSV records, generating signatures for any vulnerability that includes a GIT affected range.
  • Filtering:
    • Excludes records without GIT-based repository information.
    • Excludes Linux Kernel vulnerabilities (identified by package name or repository URL).
    • Ensures signatures are only generated for active (non-withdrawn) records.
  • Uses ThreadPoolExecutor to handle parallel I/O for fetching records from GCS and managing batch processing.
  • Enriched records are synchronized back to GCS (as the primary storage for full protos) and the Datastore index (to maintain accurate modification timestamps and status metadata).
  • Includes a --dry-run flag for testing the workflow without modifying production data.
  • Introduces modify_storage_client_adapters to configure the GCS client with an increased HTTPAdapter connection pool and retries.

Currently the crob job is scheduled daily at 9:00 AM Sydney time which we may want to adjust to run more frequent later.

@cuixq
Copy link
Copy Markdown
Contributor Author

cuixq commented Mar 25, 2026

/gemini review

@cuixq
Copy link
Copy Markdown
Contributor Author

cuixq commented Mar 25, 2026

/gemini review

@cuixq cuixq marked this pull request as ready for review March 30, 2026 03:23
@cuixq cuixq marked this pull request as draft March 30, 2026 03:46
@cuixq
Copy link
Copy Markdown
Contributor Author

cuixq commented Mar 30, 2026

/gemini review

Copy link
Copy Markdown
Member

@michaelkedar michaelkedar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks mostly good to me, I'm just a bit concerned on how gracefully errors are handled in general

@cuixq cuixq marked this pull request as ready for review March 31, 2026 23:34
michaelkedar
michaelkedar previously approved these changes Apr 1, 2026
Copy link
Copy Markdown
Contributor

@another-rex another-rex left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From our offline discussions:

  • Add a vanir generation modified date so that we can track what version of this vuln record the vanir generation is for (so we know to skip it when regening)
  • For the initial merge, let's just skip all records that already have vanir signatures, because it's going to take far too long to do. Reenable regening once we have the vanir git optimisations

Copy link
Copy Markdown
Contributor

@another-rex another-rex left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just looked at the last change, which LGTM. Just 1 question.

michaelkedar
michaelkedar previously approved these changes Apr 2, 2026
@cuixq cuixq enabled auto-merge (squash) April 2, 2026 04:31
@cuixq cuixq disabled auto-merge April 2, 2026 04:37
@cuixq cuixq enabled auto-merge (squash) April 2, 2026 04:37
@cuixq cuixq merged commit ca474bf into google:master Apr 2, 2026
21 checks passed
@cuixq cuixq deleted the vanir branch April 2, 2026 04:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants