[Feature] Materialized View DDL #18544
Conversation
8a64264 to
2d063ee
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #18544 +/- ##
============================================
+ Coverage 64.26% 64.36% +0.10%
- Complexity 1137 1302 +165
============================================
Files 3335 3355 +20
Lines 205809 207245 +1436
Branches 32106 32381 +275
============================================
+ Hits 132256 133392 +1136
- Misses 62914 63105 +191
- Partials 10639 10748 +109
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
2d063ee to
7626877
Compare
d66cc70 to
4032804
Compare
|
High-signal issue in the source-table validation path: / still accepts a raw source name when both and exist, because the resolver prefers the OFFLINE table and only rejects the source when OFFLINE is absent. That means can still be persisted for a hybrid source even though the stored is replayed later against the raw table name. The downstream broker/scheduler query can then read REALTIME rows while validation, fingerprinting, and STALE-marking checks only covered the OFFLINE half. Since realtime commits still do not notify the MV consistency manager, this can silently drift the MV. Please reject raw names that have both OFFLINE and REALTIME variants, or require an explicitly typed source table before persisting the MV. |
|
High-signal issue in the source-table validation path: |
da3a54f to
aaf07a2
Compare
xiangfu0
left a comment
There was a problem hiding this comment.
Found 1 high-signal issue; see inline comment.
b66104c to
0e60edf
Compare
0e60edf to
d79b906
Compare
…SHOW CREATE / DROP
Adds Pinot-native SQL DDL for managing materialized views end to end,
served through the same POST /sql/ddl endpoint as plain-table DDL.
Supported statements:
CREATE MATERIALIZED VIEW [IF NOT EXISTS] [db.]name [(<column list>)]
[REFRESH EVERY 'period']
PROPERTIES ('timeColumnName' = 't', 'bucketTimePeriod' = '1d', ...)
AS <Pinot SELECT>
SHOW MATERIALIZED VIEWS
SHOW CREATE MATERIALIZED VIEW [db.]name
DROP MATERIALIZED VIEW [IF EXISTS] [db.]name
CREATE MATERIALIZED VIEW supports two forms:
- Explicit column list: every column is declared in the DDL.
- Inferred column list: the column list is omitted and the storage
schema is derived from the AS <SELECT>.
Both forms produce identical materialized-view storage from the data
plane's perspective. SHOW CREATE MATERIALIZED VIEW always renders the
canonical form with an explicit column list, so the round-trip through
DDL is stable in either case.
Co-authored-by: Xiang Fu <xiangfu.1024@gmail.com>
Signed-off-by: Hongkun Xu <xuhongkun666@163.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
d79b906 to
d6a3ab2
Compare
Polish on top of the MV DDL feature commit. Summary: Simplifications: - Delete MaterializedViewSchemaInferer interface + factory; collapse the single impl into a final class with a static infer() method. - Inline MaterializedViewPropertyExtractor.isMaterializedView() to direct TableConfig.isMaterializedView() calls; delete the wrapper. - Collapse three identical IF NOT EXISTS race-recovery checks into mvIfNotExistsNoOpResponse(...) helper. - Replace MaterializedViewPropertyRouter.canonicalTaskKnobKeys() Set lookup with a single canonicalKnobName(lowerCasedKnob) helper that does both membership and canonical-cased return. Bug fixes: - Canonicalize task.MaterializedViewTask.* prefix-form keys in the router the same way bare-form keys are canonicalized, eliminating an asymmetry where 'BUCKETTIMEPERIOD' bare and 'task.MaterializedViewTask.BUCKETTIMEPERIOD' prefixed landed under different on-wire keys. - executeDdl no longer catches RuntimeException as 400. Programmer errors now propagate to JAX-RS as 500 so monitoring fires. - Wrap IllegalStateException from RequestUtils.extractAliasOrIdentifierName as DdlCompilationException in the inferer so unaliased aggregations surface as 400 with guidance, not 500. - Verify isMaterializedView() in IF NOT EXISTS race-catch paths: a concurrent plain OFFLINE table create no longer fools MV DDL into returning 200. - Restore task schedules when deleteTable rejects DROP (e.g. blocked by dependent MV), via TaskCleanupRestorer; suppress the misleading "schedules need manual restoration" hint after a successful restore. - Skip orphan MV definition znodes (TableConfig missing or non-MV) in the consistency manager's reverse-index rebuild, with WARN log; prevents ghost MV resurrection after a partial DROP. - Reject malformed/negative stalenessThresholdMs in the analyzer at CREATE time; explicit 0 (the documented "no SLO" default) is accepted. - Thread request-header database into DdlCompileContext so the MV inferer resolves FROM clauses against the operator's intended DB, not the cluster default. Tests: - Add regression test for the bare/prefix case canonicalization. - Add regression test for unaliased-aggregation DdlCompilationException. - Add regression test for the orphan-znode skip branch. - Add regression tests for stalenessThresholdMs analyzer validation (malformed, negative, explicit 0). - Add regression test for the race-catch path where a plain OFFLINE table races in under IF NOT EXISTS — must surface as 409, not 200. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Pushed Simplifications (~410 net lines removed)
Bug fixes
Tests added
All affected modules pass |
[Feature] Materialized View DDL
Summary
Adds Pinot-native SQL DDL for managing Materialized Views (MV) end-to-end, served through the existing
POST /sql/ddlcontroller endpoint introduced by the Table DDL PR. Four new statements:Before this PR, MVs could only be authored / inspected / torn down via the JSON
POST /tablespath — verbose, error-prone, and impossible to copy-paste between environments. This PR makes MVs a first-class SQL DDL surface with a deterministic round-trip:CREATE→ ZK →SHOW CREATEproduces a statement that re-parses to the sameTableConfig.Labels:
feature,release-notes(new SQL grammar + new controller error contracts),sql-compliance,materialized-view.Scope & alignment with prior PRs
This MV DDL is implemented on top of the following two PRs. Its functional scope is intentionally kept aligned with theirs — anything outside that joint baseline is out of scope here and tracked on the respective upstream layer instead.
DDL 1:
CREATE MATERIALIZED VIEWThere are two ways to author the column list.
AS-driven inference is the recommended path. Use the explicit form only if you need to override a specific column type (e.g. tighten aLONGtoINT).Two ways, same MV
The following two statements produce a byte-identical persisted
TableConfig+Schema:No mixing
The column list is all-or-nothing:
(...)column list entirely; every column is derived from theAS <query>projection.A column list that covers only some of the projection columns (or that adds extra columns the projection does not produce) is rejected at compile time with HTTP 400 and a clear pointer to either drop the column list or complete it.
Inferred type & role mapping
When the column list is omitted, each
SELECTprojection item maps to an MV column as follows:AS <query>)col(non-time identity passthrough)DIMENSION/METRIC/DATETIME)DATETIMEformat/granularity also inherited from the source schemacol AS twheret = timeColumnName(time-column passthrough)DATETIMENOT NULLDATETRUNC('unit', ts[, ...]) AS twheret = timeColumnNameTIMESTAMPDATETIME1:MILLISECONDS:TIMESTAMP, granularity derived from the truncation unit;NOT NULLts * f1 [* f2 …] AS twheret = timeColumnName(arithmetic scaling)TIMESTAMPDATETIME1:MILLISECONDS:TIMESTAMP, granularity1:MILLISECONDS;NOT NULLSUM(expr)/MIN(expr)/MAX(expr)DOUBLEMETRICNOT NULLCOUNT(expr)/COUNT(*)LONGMETRICNOT NULLDISTINCTCOUNTRAWHLL(expr)/DISTINCTCOUNTRAWHLLPLUS(expr)/DISTINCTCOUNTRAWTHETASKETCH(expr)BYTESMETRICNOT NULL. Raw sketches are stored as serialized bytes; both engines surface them as hex-encodedSTRINGto query clients, but the MV column must beBYTESfor the rewrite engine to re-aggregate them correctly.Aggregations not in the table above (e.g.
AVG,DISTINCTCOUNT, custom UDAFs) are rejected at compile time with HTTP 400 and a pointer to the supported set. Multi-value source columns are rejected (MV does not currently support multi-value).REFRESH EVERY '<period>'Optional. Registers a per-MV Quartz cron under
task.MaterializedViewTask.schedule. When omitted, the MV runs under the cluster-widePinotTaskManagerschedule.<period>is a single positive integer followed by a unit charm/h/d:'Nm''Nh''Nd'Rejected at parse / compile time (HTTP 400):
'60m','24h','29d'and above — use the next-larger unit.'0d'/'-1m'/'1.5h'— non-positive or non-integer.'1w'/'1y'/'30s'/'every Monday'— unsupported unit.SHOW CREATE MATERIALIZED VIEWon such an MV returns 400 rather than silently emitting an un-round-trippable DDL.For convenience,
EVERY { MINUTE[S] | HOUR[S] | DAY[S] }is accepted as a sugared form (e.g.REFRESH EVERY 15 MINUTES) and normalized to'15m'internally.PROPERTIESSingle flat string-to-string map. Keys are case-insensitive on input; canonical casing is restored on the wire.
Required
timeColumnNameDATETIMEcolumn produced by the projection. The analyzer rejectsLONG/INTtime columns at create time; in the inferred form, the time column must be either an identity passthrough of aTIMESTAMPsource, aDATETRUNCcall, or arithmetic-scaling.bucketTimePeriod'1h','1d','7d', …)> 0.Optional MV task knobs
All can be written either bare (preferred) or with the
task.MaterializedViewTask.prefix.SHOW CREATEalways emits the bare form.bufferTimePeriod≥ 0.maxNumRecordsPerSegment5000000maxTasksPerBatch4(hard cap1000)stalenessThresholdMs0(SLO disabled)(now - watermarkMs) > stalenessThresholdMs, the broker excludes this MV from query-rewrite.0means no check.taskModeOptional table-level promoted knobs
replicationbrokerTenantDefaultTenantserverTenantDefaultTenanttimeTypeFORMAT '..'CREATE TABLE.Rejected at compile time (HTTP 400)
schedule(bare ortask.MaterializedViewTask.schedule)REFRESH EVERY '<period>'.definedSQL(bare ortask.MaterializedViewTask.definedSQL)AS <query>clause.streamType,stream.*,realtime.*tableType,tableName,ifNotExistsLimitations enforced at compile time
AS <query>rejectsJOIN,SELECT *, and DML (INSERT/UPDATE/DELETE/MERGE)REFRESH EVERYand arbitrary cron expressionsDDL 2:
SHOW MATERIALIZED VIEWS [FROM db]Lists raw MV names (no
_OFFLINEsuffix) in the database scope. Identification is driven by the canonicalTableConfig.isMaterializedView()flag. MVs continue to appear inSHOW TABLES(they are physically OFFLINE tables);SHOW MATERIALIZED VIEWSis the type-narrowed view of the same catalog.DDL 3:
SHOW CREATE MATERIALIZED VIEW [db.]nameReturns the canonical, re-parseable DDL for the named MV. Always emits the explicit-column form for round-trip stability — the output is byte-identical regardless of whether the MV was originally created with an inferred or explicit column list.
Emission rules:
REFRESH EVERY '<period>'emitted only when the stored cron round-trips via the inverse mapping; non-standard hand-crafted cron expressions trigger HTTP 400 with a pointer back to the JSON API rather than silently dropping the schedule.task.MaterializedViewTask.definedSQLis preserved as the raw user-typed substring ofAS <query>and re-emitted verbatim — no Calcite re-rendering, no dialect drift.TYPEclause is accepted (a trailingTYPE OFFLINEfails parsing rather than being silently swallowed).emit → parse → emitis exercised end-to-end.DDL 4:
DROP MATERIALIZED VIEW [IF EXISTS] [db.]nameTears down the MV, including its definition znode, runtime watermark znode, and Helix resource. No
TYPEclause.IF EXISTSswallows a missing MV (200 no-op). It does not swallow type confusion: if the name resolves to a plain OFFLINE table, the call returns 400 pointing atDROP TABLE.More examples
Sub-hour refresh + bounded staleness SLO (inferred)
Heavy daily roll-up with aggressive back-fill and segment sizing (inferred)
POSTing DDL to the controller
Strict TABLE / MATERIALIZED VIEW partitioning at the REST boundary
The DDL verbs refuse to act on the wrong target type — enforced symmetrically:
SHOW CREATE TABLEon an MVSHOW CREATE MATERIALIZED VIEWon a plain tableDROP TABLEon an MVDROP MATERIALIZED VIEWon a plain tableDROP MATERIALIZED VIEW IF EXISTSon a plain tableIF EXISTSis about absence, not type confusion.REST endpoint contract
All MV DDL operations are sent to the same endpoint as the Table DDL PR:
Response codes (MV-specific outcomes)
200 OKIF [NOT] EXISTSno-op / dry-run201 Created400 Bad RequestAS <query>shape; unsupported aggregation; mixed (partial-explicit) column list; non-round-trippable cron onSHOW CREATE.404 Not FoundIF EXISTS, orSHOW CREATEon a missing MV).409 ConflictIF NOT EXISTS; race lost to a concurrent writer.500 Internal Server ErrorThe full response shape (
operation,tableName,databaseName,schema,tableConfig,warnings,message, …) is the same as the Table DDL PR.SHOW MATERIALIZED VIEWSpopulates the existingtableNamesfield.Dry-run
?dryRun=truecompiles + validates and returns the would-beSchema+TableConfig, but persists nothing. Useful for CI / pre-flight checks. Inferred column lists are fully resolved during dry-run, so operators can preview the inferredSchemabefore committing.Authorization
MV DDLs reuse the same action constants as their Table DDL counterparts (
CREATE_TABLE,GET_TABLE_CONFIG,DELETE_TABLE). Introducing new MV-specific actions would silently lock operators withCREATE_TABLEout of MV creation during rolling upgrade. Authorization happens after database-name translation, against the post-translation resource.Rolling-upgrade safety
The DDL feature is purely additive. No SPI signature, no enum, no
TableConfig/Schemafield, and no ZK property-store path is renamed or removed. Persisted artifacts produced byPOST /sql/ddlare shape-identical to those produced by the existingPOST /tablesandPOST /schemasendpoints — old brokers, servers, and controllers read DDL-created MVs exactly as they read JSON-API-created MVs.POST /sql/ddl. There is no half-state.MATERIALIZED,REFRESH,VIEWS) are added undernonReservedKeywordsToAdd. Existing DQL queries using them as identifiers continue to parse on the new binary.Test plan
PinotDdlParserTest) —CREATE MATERIALIZED VIEWall syntactic variants, including the optional column list (present / empty / mixed-rejection);REFRESH EVERYquoted-period and sugared (N MINUTES/N HOURS/N DAYS) forms;SHOW MATERIALIZED VIEWS [FROM db];SHOW CREATE MATERIALIZED VIEW;DROP MATERIALIZED VIEW [IF EXISTS]; rejection ofTYPEon MV-specific SHOW/DROP.DdlCompilerMaterializedViewTest,DdlCompilerMaterializedViewInferenceTest) — everyPROPERTIESrouting rule; period → Quartz cron mapping including boundary rejections;AS <query>raw-substring preservation; rejection ofJOIN,SELECT *, DML, undefined-column references; partial-column-list rejection; unsupported aggregation rejection; multi-value source column rejection.MultiStageMaterializedViewSchemaInfererTest) — every projection shape in the type/role mapping table, including all 7 supported aggregations (sketches assertBYTES), all three supported time-column expression shapes (identity passthrough,DATETRUNC, arithmetic scaling), and the corresponding negative cases.CanonicalDdlEmitterMaterializedViewTest,MaterializedViewPropertyExtractorTest) — canonical clause order; lexicographic property ordering; non-round-trippable cron rejection; MV task knob flattening;SHOW CREATEalways emits the explicit-column form regardless of how the MV was created.RoundTripTest) —emit → parse → compile → emitidempotence across all PROPERTIES routing rules + theREFRESH EVERYclause + both column-list forms.PinotDdlRestletResourceMaterializedViewUnitTest) — CREATE / SHOW CREATE / SHOW MATERIALIZED VIEWS / DROP dispatch, response shape, authorization (403 on permission deny), 400 on strict type-partitioning violations.PinotHelixResourceManagerMaterializedViewListingTest) —getAllRawMaterializedViewNamesfilters byisMaterializedView(), skips REALTIME, tolerates corrupted znodes, scopes by database.MaterializedViewDdlAnalyzerIntegrationTest) — DDL creates an MV that realMaterializedViewAnalyzeraccepts on a live cluster, end-to-end for both inferred and explicit column-list forms.All tests pass.
./mvnw spotless:apply checkstyle:check license:format license:checkclean acrosspinot-common,pinot-sql-ddl,pinot-controller, andpinot-materialized-view.