Promotion 2026-05-26 prod (#8052) by achave11-ucsc · Pull Request #8057 · DataBiosphere/azul

achave11-ucsc · 2026-05-28T04:39:46Z

Linked issue: #8052

Checklist

Author

Author (reindex)

This PR is labeled reindex:prod _{or the changes introduced by it will not require reindexing of prod}
This PR is labeled reindex:partial and its description documents the specific reindexing procedure for prod _{or requires a full reindex or is not labeledreindex:prod}

Author (mirror)

This PR is labeled mirror:prod _{or the changes introduced by it will not require mirroring of prod}
This PR is labeled mirror:partial and its description documents the specific mirroring procedure for prod _{or requires a full mirroring or is not labeledmirror:prod}

Author (upgrading deployments)

This PR is labeled upgrade _{or does not require upgrading deployments}
This PR is labeled deploy:shared _{or does not modify docker_images.json, and does not require deploying the shared component for any other reason}
This PR is labeled deploy:gitlab _{or does not require deploying the gitlab component}
This PR is labeled deploy:runner _{or does not require deploying the runner image}

Author (before every review)

PR branch is up to date (if not, merge prod into PR branch to integrate upstream changes)
PR is not a draft
PR is awaiting requested review from system administrator
Status of PR is Review requested
PR is assigned to only the system administrator and the author

System administrator (after approval)

Actually approved the PR
Labeled PR as no sandbox
N reviews label is accurate
Status of PR is Approved
PR is assigned to only the operator and the author

Operator

Pushed PR branch to GitHub

Operator (deploy `.shared` and `.gitlab` components)

Ran _select prod.shared && CI_COMMIT_REF_NAME=prod make -C terraform/shared apply_keep_unused _{or this PR is not labeled deploy:shared}
Ran _select prod.gitlab && python scripts/create_gitlab_snapshot.py --no-restart (see operator manual for details) _{or this PR is not labeled backup:gitlab}
Ran _select prod.gitlab && CI_COMMIT_REF_NAME=prod make -C terraform/gitlab apply _{or this PR is not labeled deploy:gitlab}
Checked the items in the next section _{or this PR is labeled deploy:gitlab}
PR is assigned to only the system administrator and the author _{or this PR is not labeled deploy:gitlab}

System administrator (post-deploy of `.gitlab` component)

Background migrations for prod.gitlab are complete _{or this PR is not labeled deploy:gitlab}
PR is assigned to only the operator and the author

Operator (deploy runner image)

Ran _select prod.gitlab && make -C terraform/gitlab/runner _{or this PR is not labeled deploy:runner}

Operator (merge the branch)

All status checks passed and the PR is mergeable
The title of the merge commit starts with the title of this PR
Added PR # reference to merge commit title
Collected commit title tags in merge commit title _{but excluded any p tags}
Pushed merge commit to GitHub
Status of PR is Merged stable

Operator (main build)

¹ Promoted issues and PRs are referenced in the titles of the commits
that the promotion branch introduces to the stable branch. Prior to the
promotion, the status of promoted issues (PRs) is Lower (Merged lower).
Promoted PRs in status Done do not need to be moved.

Operator (reindex)

Deindexed all unreferenced catalogs in prod _{or this PR is neither labeled reindex:partial nor reindex:prod}
Deindexed specific sources in prod _{or this PR is neither labeled reindex:partial nor reindex:prod}
Indexed specific sources in prod _{or this PR is neither labeled reindex:partial nor reindex:prod}
Started reindex in prod _{or this PR does not require reindexing prod}
Checked for, triaged and possibly requeued messages in both fail queues in prod _{or this PR does not require reindexing prod}
Emptied fail queues in prod _{or this PR does not require reindexing prod}
Restarted the Data Browser pipeline for the ucsc/hca/prod branch on GitLab in prod _{or this PR does not require reindexing prod}
Restarted the Data Browser pipeline for the ucsc/lungmap/prod branch on GitLab in prod _{or this PR does not require reindexing prod}
Restarted deploy_browser job in the GitLab pipeline for this PR in prod _{or this PR does not require reindexing prod}

Operator (mirroring)

Started mirroring in prod _{or this PR is not labelled mirror:prod}
Checked for, triaged and possibly requeued messages in mirror fail queue in prod _{or this PR is not labelled mirror:prod}
Emptied mirror fail queue in prod _{or this PR is not labelled mirror:prod}

Operator

PR is assigned to only the system administrator

System administrator

Removed unused image tags from pycharm image on DockerHub _{or this promotion does not alter references to that image}
Removed unused image tags from bigquery_emulator image on DockerHub _{or this promotion does not alter references to that image}
PR is assigned to no one

Shorthand for review comments

L line is too long
W line wrapping is wrong
Q bad quotes
F other formatting problem

)

…es (#8008, #8009, PR #8012)

The next commit adds an assertion to the mirror service that would fail during this check after the first request due to the missing `source` attribute

…ifests (#7930)

…7931)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…ifests (#7930, #7686, PR #7995)

…ineDefinition (#8006)

The `to_json`, `from_json`, `_cls_field`, `_cls_for_name`, and `__init_subclass__` methods were added in 964c6a2 to serialize `Authentication` instances into Step Functions state for curl manifest generation. That usage was removed in a subsequent refactor, leaving the methods exercised only by their own doctest. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Scaffold for minting Azul Personal Access Tokens (APATs). When a curl manifest is generated, the manifest controller attempts to mint an APAT for the authenticated user. The user service resolves the access token via Google tokeninfo endpoint to obtain the `sub` claim, then looks up the stored tokens in DynamoDB. If the user has not previously completed the authorization flow, the exception is caught and the manifest falls back to the regular access token. The actual JWT minting and KMS signing are not yet implemented. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Use a private PyJWT instance instead of the global jwt module functions, so that a custom signing algorithm can later be registered without affecting other code in the process. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Implement APAT minting with KMS-backed JWT signing. A custom PyJWT Algorithm subclass delegates RS256 signing and verification to AWS KMS. The algorithm is registered on a private PyJWT instance to avoid conflicts with the global JWT state. The minted token is verified immediately after signing to confirm the KMS round-trip. Also adds the KMS key and alias in Terraform, IAM permissions for kms:Sign/Verify/GetPublicKey, and APAT config properties. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Reduce KMS provisioning duplication (#8007) Consolidate the per-key config (name, usage, spec, alias) into a frozen KMSKey dataclass. Terraform and IAM policy now use comprehensions over Config.kms_keys instead of repeating the pattern for each key. The APAT token expiration moves to UserService since it is unrelated to the KMS key lifecycle. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Add APAT validation and token exchange to the repository controller. When a request carries a bearer token that looks like an APAT, validate it via KMS, look up the user in DynamoDB, refresh the Google access token if it's about to expire, and return the usable access token. Forged or expired APATs now yield a 401 instead of a 500. Also store access token expiration in DynamoDB to avoid a round trip to Google's tokeninfo endpoint on every request, and move the unknown-user check into get_user() to simplify callers. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

… PR #8023)

…8045)

…n non-dataset endpoints (#8035)

…n non-dataset endpoints (#8035, PR #8046)

coveralls · 2026-05-28T04:52:28Z

coverage: 84.926% (-0.2%) from 85.104% — promotions/2026-05-26-prod into prod

codecov · 2026-05-28T04:59:59Z

Codecov Report

❌ Patch coverage is 74.77243% with 194 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.84%. Comparing base (0303df5) to head (b36d2b8).

Files with missing lines	Patch %	Lines
test/integration_test.py	0.00%	133 Missing ⚠️
src/azul/service/user_controller.py	69.49%	18 Missing ⚠️
src/azul/oauth2.py	9.09%	10 Missing ⚠️
src/azul/service/user_service.py	95.90%	7 Missing ⚠️
src/azul/service/manifest_controller.py	44.44%	5 Missing ⚠️
src/azul/service/repository_controller.py	64.28%	5 Missing ⚠️
src/azul/service/index_service.py	20.00%	4 Missing ⚠️
src/azul/opensearch.py	25.00%	3 Missing ⚠️
src/azul/__init__.py	88.88%	2 Missing ⚠️
src/azul/auth.py	90.00%	2 Missing ⚠️
... and 4 more

Additional details and impacted files

@@            Coverage Diff             @@
##             prod    #8057      +/-   ##
==========================================
- Coverage   85.02%   84.84%   -0.18%     
==========================================
  Files         165      165              
  Lines       23667    24169     +502     
==========================================
+ Hits        20123    20507     +384     
- Misses       3544     3662     +118

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

nadove-ucsc and others added 30 commits May 8, 2026 15:24

Use iif in s3 template

1ed30bc

Fix: Managed-access mirror bucket lacks lifecycle resource (#8008, #8009

6f73c28

)

Fix: Managed-access mirror bucket lacks lifecycle resource (#8009)

15a951d

Fix: Managed-access mirror bucket lacks lifecycle and logging resourc…

d111363

…es (#8008, #8009, PR #8012)

Only check if files are mirrored on first request

2801fcf

The next commit adds an assertion to the mirror service that would fail during this check after the first request due to the missing `source` attribute

Encapsulate optionality of mirror URL in MirrorService

92ce508

Ensure that a file's mirror URI is correct in service responses & man…

e78a705

…ifests (#7930)

Suppress downloads from managed-access mirror bucket and add FIXME (#…

ebcc244

…7931)

Remove typing workaround for functools.cache (#7686)

760468d

Improve type hints for azul.collections.lookup (#6821)

384ea1a

Rename method, improve docstring

77f67fa

Narrow some bits in IT (#6821)

dfa490b

Thread SourceRef through to mirror_uri to avoid source=None

b935f21

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Consolidate mirroring predicates into will_mirror and may_mirror

e83d9f3

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Add FIXME (#7634)

5111b4b

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Add FIXME (#8024)

d101d3c

Ensure that a file's mirror URI is correct in service responses & man…

2be7884

…ifests (#7930, #7686, PR #7995)

Bump tempdev incarnation counter

5927ff0

Fix: Deploying from scratch causes access denied to ValidateStateMach…

dbd7afc

…ineDefinition (#8006)

Update Google TF provider to 7.31.0 (#7968)

408bacb

Update Google TF provider to 7.31.0 (#7968, #8006, PR #7997)

26a8a1c

Rename azul.auth.OAuth2 to AccessTokenAuth (#8007)

87e9d9c

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Convert Config properties to attributes (#8007)

939f7da

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Merge branch 'develop' into backports/b4edc00

d1c533e

Use a private PyJWT instance (#8007)

d1d2af7

Use a private PyJWT instance instead of the global jwt module functions, so that a custom signing algorithm can later be registered without affecting other code in the process. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

hannes-ucsc and others added 11 commits May 20, 2026 22:29

Assert ttl_attribute is consistent with User annotations

fb39124

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Builder pattern for ItemUpdate.set and set_from

c5b1c7f

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

[u a] Support per-user APAT revocation (#8033, PR #8037)

057553a

Remove outdated comments

efdc99b

[R] Fix: DeprecationWarning for datetime methods in Python 3.12 (#5953)

482bbb1

[R] Ran make requirements_update

62f0abc

[R] Fix: DeprecationWarning for datetime methods in Python 3.12 (#5953,…

b0c4feb

… PR #8023)

Fix: GH Action schedule failed to create promotion ticket (#8041)

a28f014

Fix: GH Action schedule failed to create promotion ticket (#8041, PR #…

2ce0da3

…8045)

[a] Fix: Organic dataset fields missing from inner dataset entities o…

5e3af7a

…n non-dataset endpoints (#8035)

[a] Fix: Organic dataset fields missing from inner dataset entities o…

6b3d4a8

…n non-dataset endpoints (#8035, PR #8046)

achave11-ucsc requested a review from hannes-ucsc as a code owner May 28, 2026 04:39

achave11-ucsc linked an issue May 28, 2026 that may be closed by this pull request

Promotion 2026-05-26 #8052

Open

8 tasks

github-advanced-security AI found potential problems May 28, 2026

View reviewed changes

Comment thread src/azul/service/user_service.py Dismissed

Comment thread src/azul/service/user_service.py Dismissed

Comment thread src/azul/service/user_service.py Dismissed

Comment thread src/azul/service/user_service.py Dismissed

achave11-ucsc self-assigned this May 28, 2026

achave11-ucsc added upgrade [process] PR includes commit requiring manual upgrade deploy:gitlab [process] PR requires deploying `gitlab` component deploy:shared [process] PR requires deploying `shared` component backup:gitlab [process] PR requires backing up GitLab instances labels May 28, 2026

Merge branch 'prod' into promotions/2026-05-26-prod

b36d2b8

achave11-ucsc added deploy:runner [process] PR requires deploying `runner` component API API change affecting callers reqs [process] PR includes commit requiring ``make requirements`` labels May 28, 2026

achave11-ucsc assigned hannes-ucsc May 28, 2026

hannes-ucsc approved these changes May 28, 2026

View reviewed changes

hannes-ucsc added no sandbox [process] PR will not be tested in the sandbox 0 reviews [process] Lead didn't request any changes labels May 28, 2026

hannes-ucsc removed their assignment May 28, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Promotion 2026-05-26 prod (#8052)#8057

Promotion 2026-05-26 prod (#8052)#8057
achave11-ucsc wants to merge 103 commits into
prodfrom
promotions/2026-05-26-prod

achave11-ucsc commented May 28, 2026 •

edited by hannes-ucsc

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

coveralls commented May 28, 2026 •

edited

Loading

Uh oh!

codecov Bot commented May 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

achave11-ucsc commented May 28, 2026 • edited by hannes-ucsc Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Author

Author (reindex)

Author (mirror)

Author (upgrading deployments)

Author (before every review)

System administrator (after approval)

Operator

Operator (deploy .shared and .gitlab components)

System administrator (post-deploy of .gitlab component)

Operator (deploy runner image)

Operator (merge the branch)

Operator (main build)

Operator (reindex)

Operator (mirroring)

Operator

System administrator

Shorthand for review comments

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

coveralls commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented May 28, 2026

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

achave11-ucsc commented May 28, 2026 •

edited by hannes-ucsc

Loading

Operator (deploy `.shared` and `.gitlab` components)

System administrator (post-deploy of `.gitlab` component)

coveralls commented May 28, 2026 •

edited

Loading