roachtest: GCS failover for oversized artifacts.zip#170147
Open
williamchoe3 wants to merge 2 commits into
Open
roachtest: GCS failover for oversized artifacts.zip#170147williamchoe3 wants to merge 2 commits into
williamchoe3 wants to merge 2 commits into
Conversation
Contributor
|
Merging to
After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here |
Member
|
This change is |
williamchoe3
force-pushed
the
roachtest-artifact-failover
branch
from
06b04f9 to
3b4590b
Compare
Contributor
Author
|
Assumes the invoking script process already has GCP credentials available which is set TODO:
|
williamchoe3
force-pushed
the
roachtest-artifact-failover
branch
7 times, most recently
from
65f0581 to
33a2d03
Compare
Contributor
Author
|
|
williamchoe3
force-pushed
the
roachtest-artifact-failover
branch
from
33a2d03 to
d30e323
Compare
williamchoe3
changed the title
williamchoe3
force-pushed
the
roachtest-artifact-failover
branch
2 times, most recently
from
b295625 to
e925dee
Compare
Contributor
Author
williamchoe3
force-pushed
the
roachtest-artifact-failover
branch
2 times, most recently
from
e8bbdef to
ced67a1
Compare
Contributor
Author
williamchoe3
requested review from
golgeek and
shailendra-patel
and removed request for
a team
rickystewart
approved these changes
May 11, 2026
williamchoe3
force-pushed
the
roachtest-artifact-failover
branch
from
6e7bb54 to
6160346
Compare
Roachtest currently publishes artifacts.zip with TeamCity's publishArtifacts service message, which has a 7 GB limit. Some runs now produce artifacts.zip files above 20 GB, causing TeamCity artifact publication to fail. Add a GCS failover path for artifacts.zip files at or above 7 GB. Instead of uploading the large artifacts.zip to TeamCity, roachtest uploads it to GCS and publishes a small artifacts-failover.txt breadcrumb file pointing to the GCS URI. Release note: None
Release note: None
williamchoe3
force-pushed
the
roachtest-artifact-failover
branch
from
6160346 to
79390b3
Compare
williamchoe3
commented
May 11, 2026
| // has completed. We're using the exact same destination to avoid | ||
| // duplication of any of the artifacts. | ||
| shout(ctx, l, stdout, "##teamcity[publishArtifacts '%s']", t.artifactsSpec) | ||
| publishTeamCityArtifacts := func() { |
Contributor
Author
There was a problem hiding this comment.
I could move this into a helper, runTest was already really long before this though (350 lines). Can go either way
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Previously,
artifacts.zipwas uploaded via TeamCity service message"##teamcity[publishArtifacts '%s']", t.artifactsSpec). The current size limit for this upload method is 7GB. We have started to observe some tests attempting to uploadartifacts.zipthat is way beyond that limit i.e. 20+GB.This change adds a GCS failover mechanism to roachtest that will trigger when
artifacts.zipis >= 7GB. Then, instead of the uploading the artifact to TC, it is uploaded to GCS instead. Then in the large artifact's place on TC, a small breadcrumb fileartifacts-failover.txtis added that points to the artifact URI on GCS. See below for an example. A github issue for a failed test that had it's artifacts uploaded to GCS will then have the link to this breadcrumb file. Currently a timeout of 2 hours is used. The failover threshold and bucket default to production values, but can be overridden for smoke tests withROACHTEST_ARTIFACT_FAILOVER_MAX_BYTESandROACHTEST_ARTIFACT_FAILOVER_BUCKET. Furthermore, GCS object names are build and test scoped to avoid collisions.Alternatively, we could simply increase the TeamCity upload limit, but this particular setting is a global setting that would apply to all projects. Also we currently do not know what the maximum artifact upload size currently is / will be in the future, so if we do rely on that setting, we would have to arbitrarily guess and then adjust as we need to.
Epic: none