[catalog] convert mz_sources table to materialized view#36674
Open
mtabebe wants to merge 1 commit into
Open
Conversation
mtabebe
force-pushed
the
ma/convert-builtins/mz-sources
branch
from
9e1f06d to
a5483b4
Compare
Problem: mz_sources was a builtin table populated imperatively in Rust code in builtin_table_updates.rs. Ideologically we want to move these to views over the catalog. Solution: Convert mz_sources into a materialized view that reads directly from mz_catalog_raw. User sources are derived by parsing their stored create_sql. Builtin and log sources, which don't appear in mz_catalog_raw, are exposed via a new internal view mz_internal.mz_builtin_sources Testing: - Updated information_schema_tables.slt to expect MATERIALIZED VIEW for mz_sources - Added mz_builtin_sources to mz_internal.slt and mz_catalog_server_index_accounting.slt - updated catalog.td to move mz_sources from the tables section into the materialized views section. Part of SQL-118. Port of MaterializeInc#36136 (adapted for the split builtin file structure and current OID range).
mtabebe
force-pushed
the
ma/convert-builtins/mz-sources
branch
from
a5483b4 to
b17477f
Compare
mtabebe
requested review from
SangJunBak and
aljoscha
and removed request for
a team
def-
requested changes
May 22, 2026
Contributor
def-
left a comment
def-
left a comment
There was a problem hiding this comment.
Do we intentionally remove the progress subsource type? If so we should remove it from the documentation too. If not, we should keep it.
Full nightly will be required for this PR, triggered: https://buildkite.com/materialize/nightly/builds/16516 Lots of related failures, both about progress type and the now missing cluster_id
| "mz_secrets", | ||
| ), | ||
| MigrationStep::replacement( | ||
| "26.26.0-dev.0", |
Contributor
There was a problem hiding this comment.
Should be 26.27.0-dev.0 I assume
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem:
mz_sources was a builtin table populated imperatively in Rust code in builtin_table_updates.rs.
Ideologically we want to move these to views over the catalog.
Solution:
Convert mz_sources into a materialized view that reads directly from mz_catalog_raw. User sources are derived by parsing their stored create_sql. Builtin and log sources, which don't appear in mz_catalog_raw, are exposed via a new internal view mz_internal.mz_builtin_sources
Testing:
Part of SQL-118.
Port of #36136 (adapted for the split builtin file structure and current OID range).