Skip to content

feat(sea): per-statement metricViewMetadata knob on ExecuteStatementOptions#397

Open
msrathore-db wants to merge 2 commits into
mainfrom
msrathore/sea-parity-metric-view-metadata
Open

feat(sea): per-statement metricViewMetadata knob on ExecuteStatementOptions#397
msrathore-db wants to merge 2 commits into
mainfrom
msrathore/sea-parity-metric-view-metadata

Conversation

@msrathore-db
Copy link
Copy Markdown
Contributor

Summary

Adds an explicit metricViewMetadata?: boolean knob to ExecuteStatementOptions. When true, the Thrift backend forwards spark.databricks.optimizer.enableMetricViewMetadata=true via the outgoing TExecuteStatementReq.confOverlay, scoped to a single statement.

This is the C7 (metric-view-metadata) cluster of the autonomous Thrift↔SEA parity drive — audit refs:

  • Row 1.17 of sea-workflow/audits/2026-05-28-cross-driver-audit.md — Path forward: "explicit metricViewMetadata knob on ExecuteStatementOptions (not just session-level)."
  • Row 2.18 — F12 in the PR Bump proxy-agent from 6.5.0 to 7.0.0 #347 audit.

What changed

  • lib/contracts/IDBSQLSession.ts: new metricViewMetadata?: boolean field on ExecuteStatementOptions with JSDoc explaining the per-statement scope and the SEA wiring gap.
  • lib/DBSQLSession.ts: when the option is true, merge the Spark conf key into request.confOverlay alongside any query_tags already serialized there.
  • tests/unit/DBSQLSession.test.ts: four unit tests covering set / unset / explicit-false / coexists-with-queryTags.

SEA wiring

The SEA backend will route the same key through napi statementConf once the kernel statement-options surface lands (kernel PR #75 + NodeJS PR #393). Until then the option is honored only on Thrift; the public JSDoc states this so users do not silently lose the conf on SEA. Cross-driver round-trip is exercised in the companion driver-test PR.

Test plan

  • npm test -- tests/unit/DBSQLSession.test.ts — all 4 new tests pass; the only failing test (canDecompressLZ4Result at line 235) is a pre-existing failure on origin/main, unrelated to this change.
  • npm run lint — clean.
  • Live warehouse round-trip — covered in the companion driver-test PR (databricks/databricks-driver-test#NNN), where BackendComparator.forEachBackend exercises both Thrift and SEA legs against the pecotesting warehouse with DATABRICKS_PECOTESTING_TOKEN_PERSONAL.
  • Cross-driver validation — Python python-sea-oracle agent has been asked to verify statement_conf={"spark.databricks.optimizer.enableMetricViewMetadata": "true"} round-trips on use_sea=True; reply pending.

Notes

  • DCO sign-off + Co-authored-by: Isaac trailer on the commit.
  • Per-statement scope contract: setting the option on one execute() must NOT leak to the session. The companion driver-test asserts this by checking SET <key> on the same session after the call.

…ntOptions

Add `metricViewMetadata?: boolean` to `ExecuteStatementOptions`. When true,
the Thrift backend forwards
`spark.databricks.optimizer.enableMetricViewMetadata=true` via the request
`confOverlay`, alongside any `query_tags` already serialized there. The
option is per-statement only and does not persist across queries.

The SEA backend will route the same key through napi `statementConf` once
the kernel statement-options surface lands; until then the option is
honored only on Thrift. Documented in the public option JSDoc so users do
not silently lose the conf on SEA.

Unit tests assert the option appears in the outgoing `TExecuteStatementReq`
when set, is omitted when unset or `false`, and coexists with `queryTags`
in the same `confOverlay`.

Audit refs: rows 1.17 and 2.18 of
sea-workflow/audits/2026-05-28-cross-driver-audit.md (F12 in the PR #347
audit).

Co-authored-by: Isaac
Signed-off-by: Madhavendra Rathore <madhavendra.rathore@databricks.com>
@msrathore-db
Copy link
Copy Markdown
Contributor Author

Independent devils-advocate self-review (cold-eye, no author context)

Verdict: no HIGH or CRITICAL findings; merge-ready pending human reviewer sign-off. Two LOW notes captured for transparency:

L1 (semantic — accept-as-designed): metricViewMetadata: false is indistinguishable from omitting the option — both leave confOverlay untouched. This is intentional (the knob is opt-in only; there is no use case for explicitly clearing a per-statement conf since the default is unset). Worth a one-line JSDoc clarification if a follow-up touches this option, but not a merge blocker.

L2 (forward-reference): The JSDoc mentions "this knob is honored only on the Thrift backend", which forward-references the SEA backend that does not exist on main yet (it's on the abstraction stack). A reviewer who greps lib/ for SEA will find nothing. The phrasing is correct once the SEA stack lands and is the only way to document the gap pre-emptively, so it stays as-is.

Verified against PR-A diff at head: pattern existence, test count (4/4 new tests pass), order-of-operations w.r.t. existing query_tags block, protocol-version gating parity with the query_tags precedent, additive-only public-surface change. No backward-compat breakage. Hardcoded conf-key string matches existing project style (same as query_tags).

Cross-checks performed: lint clean, type-check clean (pre-existing failures in examples/tokenFederation/ only), the single failing test on tests/unit/DBSQLSession.test.ts:235 is a pre-existing LZ4 cloud-fetch issue on origin/main unrelated to this change.

This pull request and its description were written by Isaac.

Address DA review note N4 on PR #397: the public type is `boolean` (not
`true`-literal), but the wiring treats `false` and omitted identically.
Spell that out in the JSDoc so callers don't expect `false` to clear a
server-side default.

Co-authored-by: Isaac
Signed-off-by: Madhavendra Rathore <madhavendra.rathore@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant