New workflow for updating schema when rescoring metrics #464
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jgraham
changed the title
Member
Author
|
This currently depends on a lot of smaller PRs that are open; once they are merged I can rebase this one and just have the features it adds. |
ksy36
approved these changes
Jan 8, 2026
Contributor
|
fyi We migrated docker-etl to GHA, so you should now be able to merge the PRs yourself without the deploy steps failing. |
Contributor
|
Also since it seems related, there is a metric-hub with metric definitions, which look similarish to some definitions in this PR: https://github.com/mozilla/metric-hub |
Rather than being part of the ETL. These are still rather broken after the schema data changes, but this is the first step in refactoring them to work in the new world.
Now that we have the schemas scored in-tree we don't want to directly deploy tables for prospective rescores. Instead the approach we take is to make the metric-rescore command do two things: * Create templates for the post-rescore scored_site_reports, any routines that need to be updated as part of the rescore, and a template for a delta view that selects bugs that are changed by the rescore. * Replace: update the scored site reports templates with the new version from a rescore, and move the current version to an archive dataset, and similarly for routines modified by the rescore. The idea is that one first deploys the new scored site reports, to allow the change to be audited, and then deploys the change that replaces the original scored site reports with the new one once the rescore is ready to go.
This means that the `webcompat-site-rank-update` command will automatically generate the right schema templates rather than actually trying ot update prod.
If we deploy a rescore we want to first record the delta between the old scores and the new scores. This is a bit tricky because we want the update to the templates to go through the PR process and be deployed by the ETL, but we want to record the rescores before we actually update the schema. To solve this we add an extra lifecycle point in the schema update for tasks that run before deploying the schema. Here we call into metric_rescore to check for a metric rescore that a) isn't marked as staging and b) isn't deployed. If that's found we assume we're about to deploy it and record the score updates resulting from that change to the metric. This isn't a perfect system since the changes aren't at all atomic (e.g. the rescore could succeed, but the deploy fail), but it's probably good enough for something we do relatively rarely.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Checklist for reviewer:
referenced, the pull request should include the bug number in the title)
.circleci/config.yml) will cause environment variables (particularlycredentials) to be exposed in test logs
telemetry-airflow
responsibly.
Note for deployments: In order to push images built by this PR, the user who merges the PR
must be in the telemetry Github team.
This is because deploys depend on the
data-eng-airflow-gcr CircleCI context.
See DENG-8850 for additional discussion.