Track bundle deploy state file sizes in telemetry#5180
Draft
shreyas-goenka wants to merge 1 commit intodatabricks:mainfrom
Draft
Track bundle deploy state file sizes in telemetry#5180shreyas-goenka wants to merge 1 commit intodatabricks:mainfrom
shreyas-goenka wants to merge 1 commit intodatabricks:mainfrom
Conversation
shreyas-goenka
force-pushed
the
shreyas-goenka/state-size-telemetry
branch
2 times, most recently
from
a81fff3 to
8a52a00
Compare
Adds a new typed BundleResourcesMetadata struct under
BundleDeployExperimental, capturing per-resource-type metadata for a
bundle deploy:
- count of resources of each type declared in the bundle configuration
- max, mean, median state size in bytes across resources of that type
- whole state file size on disk
- deployment engine ("direct" or "terraform")
For Terraform deployments the tfstate is translated to the direct-
engine representation (via the existing TerraformToGroupName map) before
sizing so per-type stats are comparable across engines.
The new count field replaces the deprecated DatabricksBundleDeployEvent
.resource_*_count fields; both are populated during the transition.
The Go mirror marks the deprecated Resource*Count fields with a
"// Deprecated:" comment.
Measurement is performed at telemetry-emission time by reading the
on-disk state file once, so this lands as a single isolated module
(bundle/phases/resources_metadata.go) with one new line at the call
site — no instrumentation in deploy mutators, state-mgmt code, or
bundle.Metrics. To remove: delete the new module and revert one line
in telemetry.go plus the proto/Go field.
Requires the new resources_metadata field on BundleDeployExperimental
from the universe PR. Lumberjack drops unknown fields, so the two PRs
can land in either order.
shreyas-goenka
force-pushed
the
shreyas-goenka/state-size-telemetry
branch
from
8a52a00 to
2ddb2e5
Compare
|
An authorized user can trigger integration tests manually by following the instructions below: Trigger: Inputs:
Checks will be approved automatically on success. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a new typed
BundleResourcesMetadatastruct underBundleDeployExperimentalcapturing per-resource-type metadata for a bundle deploy:countof resources of each type declared in the bundle configuration (replaces the deprecatedresource_*_countfields)For Terraform deploys, the tfstate is translated to the direct-engine representation before sizing so per-type stats are comparable across engines.
Companion proto PR: https://github.com/databricks-eng/universe/pull/1892380
The deprecated
Resource*CountGo fields keep being populated during the transition and are marked// Deprecated:. Measurement is isolated inbundle/phases/resources_metadata.go(one read of the on-disk state file at telemetry-emission time). One-line wiring intelemetry.go. To remove: delete the new module + revert the call site + revert the proto/Go field.This pull request and its description were written by Isaac.