Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
source: src/adapter/src/catalog/builtin_table_updates.rs
revision: 9a225279d0
revision: 90a38f32be
---

# adapter::catalog::builtin_table_updates
Expand All @@ -9,4 +9,5 @@ Generates the row-level diffs that keep the `mz_catalog` system tables (e.g. `mz
For each catalog object type, a corresponding `pack_*` function converts a `CatalogEntry` or related struct into one or more `BuiltinTableUpdate` rows with `+1` or `-1` diffs; the coordinator calls these from the DDL path.
`BuiltinTableUpdate` wraps either a `Row`-level update or a `ProtoBatch` for bulk ingestion, both resolved to a `CatalogItemId` before being written.
`pack_role_update` derives `rolcanlogin` from `role.attributes.login` (defaulting to `true` for built-in super roles) and `rolsuper` from `role.attributes.superuser` (defaulting to `true` for built-in super roles, `Datum::Null` for indeterminate cloud environments, or `false` otherwise); it also emits per-variable rows into `mz_role_parameters`.
`pack_role_auth_update` writes a row into `mz_role_auth` containing the role's password hash and the timestamp it was last updated.
The `notice` sub-module handles the special `mz_recent_activity_log` and related notice tables separately.
4 changes: 3 additions & 1 deletion doc/developer/generated/adapter/catalog/open.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,13 @@
---
source: src/adapter/src/catalog/open.rs
revision: a50264ee3d
revision: 210c00eb3d
---

# adapter::catalog::open

Implements `Catalog::open`, the full catalog initialisation sequence run at environment startup.
The function opens the durable catalog store, runs AST migrations (`catalog::migrate`), applies all persisted `StateUpdate` diffs to build the in-memory `CatalogState`, bootstraps built-in objects (schemas, roles, clusters, tables, views, materialized views, functions), runs builtin-schema migrations (see `open::builtin_schema_migration`), and returns a ready `Catalog` along with the initial builtin-table updates to write.
`add_new_remove_old_builtin_items_migration` handles `Builtin::MaterializedView` for column comments.
`add_new_remove_old_builtin_clusters_migration` emits audit log events when creating or dropping builtin clusters so those operations are visible in `mz_audit_events`.
`add_new_remove_old_builtin_cluster_replicas_migration` emits audit log events when creating or dropping builtin cluster replicas so those operations are visible in `mz_audit_events`.
`remove_pending_cluster_replicas_migration` emits audit log events when dropping pending replicas so the drops are visible in `mz_audit_events`.
4 changes: 2 additions & 2 deletions doc/developer/generated/adapter/catalog/open/_module.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
---
source: src/adapter/src/catalog/open.rs
revision: a50264ee3d
revision: 210c00eb3d
---

# adapter::catalog::open

Houses the catalog open sequence and its sub-step `builtin_schema_migration`.
The parent file implements `Catalog::open`; the `builtin_schema_migration` child handles schema migrations for builtin storage collections whose persist shard schemas change between versions; and `builtin_schema_migration_tests` provides integration test coverage.
The open sequence supports builtin materialized views (column comments, descriptor validation) and emits audit log events when removing pending cluster replicas.
The open sequence supports builtin materialized views (column comments, descriptor validation) and emits audit log events when creating or removing builtin clusters, when creating or removing builtin cluster replicas, and when removing pending cluster replicas.
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
---
source: src/adapter/src/catalog/open/builtin_schema_migration.rs
revision: f75acad2d2
revision: 8dfe3daa00
---

# adapter::catalog::open::builtin_schema_migration

Implements schema migration for builtin storage collections whose persist shard schemas change between versions.
Two migration mechanisms are provided: `Mechanism::Evolution` uses persist's schema evolution to evolve a shard in-place (backward-compatible changes only), and `Mechanism::Replacement` creates a new shard (works for all changes but discards data).
The `MIGRATIONS` list declares per-version migration steps; `Migration::run` applies outstanding steps, coordinating shard creation, schema registration, fingerprint updates, and cleanup across 0dt-compatible read-only and leader environments.
`CatalogItemType::MaterializedView` is supported alongside `Table`, `Source`, and `ContinualTask` as a valid target for schema migrations, as reflected by recent entries in the `MIGRATIONS` list that migrate several `mz_catalog` and `mz_internal` materialized views.
`CatalogItemType::MaterializedView` is supported alongside `Table`, `Source`, and `ContinualTask` as a valid target for schema migrations, as reflected by entries in the `MIGRATIONS` list that migrate several `mz_catalog` and `mz_internal` materialized views including `mz_databases`, `mz_schemas`, `mz_role_members`, `mz_network_policies`, `mz_network_policy_rules`, `mz_cluster_workload_classes`, `mz_internal_cluster_replicas`, `mz_pending_cluster_replicas`, and `mz_materialized_views`.
4 changes: 3 additions & 1 deletion doc/developer/generated/adapter/config/frontend.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
---
source: src/adapter/src/config/frontend.rs
revision: cb7bca9662
revision: 4bdebeddb9
---

# adapter::config::frontend

Defines `SystemParameterFrontend`, which pulls `SynchronizedParameters` from either a LaunchDarkly SDK client or a local JSON config file, depending on `SystemParameterSyncClientConfig`.
The frontend maps LaunchDarkly feature-flag keys to system variable names using `key_map`, then updates the `SynchronizedParameters` set with fresh values on each `pull` call.
The `SystemParameterFrontendClient` enum encapsulates the two backend variants: `LaunchDarkly` (holding an `ld::Client` and a scoped `ld::Context`) and `File` (holding a path to a local JSON config file).
`MetricsTransport<T>` wraps an `HttpTransport` implementation and records a Unix-second timestamp on each successful HTTP response into a `UIntGauge`; two instances are used — one for the event processor (CSE metric) and one for the streaming data source (SSE metric) — to track LaunchDarkly connectivity health.
3 changes: 2 additions & 1 deletion doc/developer/generated/catalog/builtin/_module.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,13 @@
---
source: src/catalog/src/builtin.rs
revision: 22c58f7c88
revision: 1ef00897b8
---

# catalog::builtin

Defines every built-in catalog object hardcoded into Materialize: system tables (`BuiltinTable`), views (`BuiltinView`), materialized views (`BuiltinMaterializedView`), indexes (`BuiltinIndex`), types (`BuiltinType`), sources (`BuiltinSource`), logs (`BuiltinLog`), continual tasks (`BuiltinContinualTask`), connections (`BuiltinConnection`), cluster configs, roles, and schemas.
Key types include `Builtin<T>` (a generic enum with variants Log, Table, View, MaterializedView, Type, Func, Source, ContinualTask, Index, and Connection, each wrapping the corresponding builtin struct), `BUILTINS` (the exhaustive static list used at catalog open time), and constants like `BUILTIN_PREFIXES` and `RUNTIME_ALTERABLE_FINGERPRINT_SENTINEL`.
`MZ_DATABASES` is a `BuiltinMaterializedView` backed by a query over the catalog.
The `builtin` submodule generates the `mz_builtin_materialized_views` view, which exposes metadata about all builtin materialized views.
The `notice` submodule contains the optimizer-notice tables.
All built-in objects are auto-installed on catalog open; changes to their definitions are detected via fingerprinting and migrated automatically.
12 changes: 12 additions & 0 deletions doc/developer/generated/catalog/builtin/builtin.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
---
source: src/catalog/src/builtin/builtin.rs
revision: 90a38f32be
---

# catalog::builtin::builtin

Generates builtin views that expose information about builtin catalog objects.

The public entry point is `builtins(builtin_items)`, which iterates over a slice of `Builtin<NameReference>` values, filters for `BuiltinMaterializedView` variants, and produces additional `Builtin::View` entries to be included in the static builtin list.

The primary output is the `mz_builtin_materialized_views` builtin view (in `mz_internal`), which reports every builtin materialized view with columns: `oid`, `schema_name`, `name`, `cluster_name`, `definition`, `privileges`, and `create_sql`. Its SQL is constructed dynamically by serializing each `BuiltinMaterializedView`'s metadata into a `VALUES` clause.
3 changes: 2 additions & 1 deletion doc/developer/generated/catalog/durable/transaction.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
---
source: src/catalog/src/durable/transaction.rs
revision: dc210976a0
revision: 210c00eb3d
---

# catalog::durable::transaction

Defines `Transaction`, which batches multiple catalog mutations and commits them atomically to the durable store.
Internally holds per-collection `TableTransaction` instances for databases, schemas, items, comments, roles, role_auth, clusters, cluster replicas, introspection sources, ID allocators, configs, settings, system GID mappings, system configurations, default privileges, source references, system privileges, storage collection metadata, unfinalized shards, txn-wal shards, and network policies.
Provides methods for every catalog DDL operation (create/update/delete for clusters, replicas, items, roles, schemas, comments, network policies, storage metadata, etc.) as well as ID allocation from the per-collection allocators.
`insert_system_cluster` returns the allocated `ClusterId` on success.
`TransactionBatch` is the serializable representation of the accumulated changes; it is passed to `DurableCatalogState::commit_transaction` for atomic persist writes.
3 changes: 2 additions & 1 deletion doc/developer/generated/catalog/memory/objects.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
source: src/catalog/src/memory/objects.rs
revision: 5680493e7d
revision: 0c2379c49c
---

# catalog::memory::objects
Expand All @@ -9,4 +9,5 @@ Defines all in-memory catalog object types, which are richer and more consumer-f
Key types include `CatalogItem` (an enum over Table, Source, Log, View, MaterializedView, Sink, Index, Type, Func, Secret, Connection, ContinualTask), `CatalogEntry` (pairs a `CatalogItem` with its metadata), `Database`, `Schema`, `Role`, `Cluster`, `ClusterReplica`, and `NetworkPolicy`.
`StateUpdate` and `StateUpdateKind` represent in-memory deltas applied during catalog replay and incremental updates.
`DataSourceDesc` and `DataSource` describe how a source or table obtains its data (ingestion, introspection, webhook, progress, etc.).
`Sink::envelope` returns `Some("append")` for the `SinkEnvelope::Append` variant in addition to `Some("debezium")` and `Some("upsert")`.
The `UpdateFrom` trait drives the incremental update pattern throughout this module.
2 changes: 1 addition & 1 deletion doc/developer/generated/ccsr/tls.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
source: src/ccsr/src/tls.rs
revision: ca42a663e0
revision: 16c15ae741
---

# mz-ccsr::tls
Expand Down
4 changes: 2 additions & 2 deletions doc/developer/generated/compute-types/dyncfgs.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
---
source: src/compute-types/src/dyncfgs.rs
revision: 7411afe76f
revision: ef950e9dbe
---

# compute-types::dyncfgs

Declares all dynamic configuration (`dyncfg`) constants for the compute layer. Key groups include:

- **Rendering**: `ENABLE_MZ_JOIN_CORE`, `LINEAR_JOIN_YIELDING`, `ENABLE_TEMPORAL_BUCKETING`/`TEMPORAL_BUCKETING_SUMMARY`, `COMPUTE_FLAT_MAP_FUEL`, `ENABLE_COMPUTE_RENDER_FUELED_AS_SPECIFIC_COLLECTION`, `COMPUTE_APPLY_COLUMN_DEMANDS`.
- **Rendering**: `ENABLE_HALF_JOIN2`, `ENABLE_MZ_JOIN_CORE`, `LINEAR_JOIN_YIELDING`, `ENABLE_TEMPORAL_BUCKETING`/`TEMPORAL_BUCKETING_SUMMARY`, `COMPUTE_FLAT_MAP_FUEL`, `ENABLE_COMPUTE_RENDER_FUELED_AS_SPECIFIC_COLLECTION`, `COMPUTE_APPLY_COLUMN_DEMANDS`.
- **MV sink correction**: `ENABLE_CORRECTION_V2`, `CONSOLIDATING_VEC_GROWTH_DAMPENER`, `MV_SINK_ADVANCE_PERSIST_FRONTIERS`.
- **Memory management**: `ENABLE_LGALLOC`, `ENABLE_LGALLOC_EAGER_RECLAMATION`, `LGALLOC_BACKGROUND_INTERVAL`, `LGALLOC_FILE_GROWTH_DAMPENER`, `LGALLOC_LOCAL_BUFFER_BYTES`, `LGALLOC_SLOW_CLEAR_BYTES`, `ENABLE_COLUMNATION_LGALLOC`, `ENABLE_COLUMNAR_LGALLOC`, `MEMORY_LIMITER_INTERVAL`, `MEMORY_LIMITER_USAGE_BIAS`, `MEMORY_LIMITER_BURST_FACTOR`.
- **Backpressure**: `DATAFLOW_MAX_INFLIGHT_BYTES`, `DATAFLOW_MAX_INFLIGHT_BYTES_CC`, `ENABLE_COMPUTE_LOGICAL_BACKPRESSURE`, `COMPUTE_LOGICAL_BACKPRESSURE_MAX_RETAINED_CAPABILITIES`, `COMPUTE_LOGICAL_BACKPRESSURE_INFLIGHT_SLACK`.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
source: src/compute/src/extensions/temporal_bucket.rs
revision: c642b63c77
revision: f498b6e141
---

# mz-compute::extensions::temporal_bucket
Expand Down
4 changes: 3 additions & 1 deletion doc/developer/generated/compute/render/join/delta_join.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,12 @@
---
source: src/compute/src/render/join/delta_join.rs
revision: e79a6d96d9
revision: ef950e9dbe
---

# mz-compute::render::join::delta_join

Renders `JoinPlan::Delta` — a join plan where each input drives a separate dataflow path that probes all other inputs via arranged lookups, and the results are unioned.
Delta joins avoid re-processing historical data when new updates arrive on one input: only the delta of the updated input flows through all probe paths.
Each path applies a sequence of `MirScalarExpr`-based filter/project steps and optionally closes off at an `until` frontier to bound output times.

The half-join step is dispatched to one of two implementations based on the `ENABLE_HALF_JOIN2` dynamic config flag: `build_halfjoin2` (the default, using `differential_dogs3::operators::half_join2`) avoids quadratic behavior in certain join patterns, while `build_halfjoin1` (the fallback) uses the original `differential_dogs3::operators::half_join`.
4 changes: 2 additions & 2 deletions doc/developer/generated/compute/row_spine.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
---
source: src/compute/src/row_spine.rs
revision: db271c31b1
revision: f498b6e141
---

# mz-compute::row_spine

Provides `Row`-specialized differential dataflow spine types (`RowRowSpine`, `RowValSpine`, `RowSpine`) with custom layouts that use `DatumContainer` for key storage.
`DatumContainer` is a dictionary-compressed container that stores rows as raw bytes and exposes them as lazy `DatumSeq` iterators, avoiding full `Row` allocation on read.
`DatumContainer` is a container that stores rows as raw bytes (backed by `BytesContainer`) and exposes them as lazy `DatumSeq` iterators, avoiding full `Row` allocation on read.
`OffsetOptimized` is a compact offset list that first tries to store entries as a stride pattern before spilling to a general `OffsetList`, reducing memory overhead for uniform-length rows.
2 changes: 1 addition & 1 deletion doc/developer/generated/compute/sink/correction_v2.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
source: src/compute/src/sink/correction_v2.rs
revision: b7426255ce
revision: f498b6e141
---

# mz-compute::sink::correction_v2
Expand Down
2 changes: 1 addition & 1 deletion doc/developer/generated/compute/typedefs.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
source: src/compute/src/typedefs.rs
revision: c642b63c77
revision: f498b6e141
---

# mz-compute::typedefs
Expand Down
2 changes: 1 addition & 1 deletion doc/developer/generated/dyncfg-launchdarkly/_crate.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
source: src/dyncfg-launchdarkly/src/lib.rs
revision: f8348f3bca
revision: 4bdebeddb9
---

# mz-dyncfg-launchdarkly
Expand Down
2 changes: 1 addition & 1 deletion doc/developer/generated/expr/explain/text.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
source: src/expr/src/explain/text.rs
revision: de1872534e
revision: 1c2d0f12f2
---

# mz-expr::explain::text
Expand Down
6 changes: 3 additions & 3 deletions doc/developer/generated/expr/scalar/_module.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
---
source: src/expr/src/scalar.rs
revision: af9155582e
revision: 90a38f32be
---

# mz-expr::scalar

Defines `MirScalarExpr`, the central scalar expression type in Materialize's MIR (Mid-level Intermediate Representation).
`MirScalarExpr` is a recursive enum with variants `Column`, `Literal`, `CallUnmaterializable`, `CallUnary`, `CallBinary`, `CallVariadic`, and `If`; it implements `VisitChildren` enabling the generic traversal infrastructure from `visit.rs`.
Also defines `EvalError` (the comprehensive error enum for runtime scalar evaluation failures, including `InvalidCatalogJson` for malformed catalog JSON), `AggregateFunc`, `AggregateExpr`, `ColumnOrder`, and associated type-inference and simplification methods on `MirScalarExpr`.
`MirScalarExpr` is a recursive enum with variants `Column` (carrying an optional name via `TreatAsEqual<Option<Arc<str>>>`), `Literal`, `CallUnmaterializable`, `CallUnary`, `CallBinary`, `CallVariadic`, and `If`; it implements `VisitChildren` enabling the generic traversal infrastructure from `visit.rs`.
Also defines `EvalError` (the comprehensive error enum for runtime scalar evaluation failures, including `InvalidCatalogJson` for malformed catalog JSON) and associated type-inference and simplification methods on `MirScalarExpr`.
Child modules: `func` provides all the concrete function implementations; `like_pattern` provides LIKE matching logic.
2 changes: 1 addition & 1 deletion doc/developer/generated/expr/scalar/func/_module.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
source: src/expr/src/scalar/func.rs
revision: fe91a762d1
revision: 90a38f32be
---

# mz-expr::scalar::func
Expand Down
Original file line number Diff line number Diff line change
@@ -1,12 +1,14 @@
---
source: src/expr/src/scalar/func/impls/case_literal.rs
revision: de1872534e
revision: 1c2d0f12f2
---

# mz-expr::scalar::func::impls::case_literal

Provides a lookup-based evaluation of `CASE expr WHEN lit1 THEN res1 ... ELSE els END`.
`CaseLiteral` replaces chains of `If(Eq(expr, literal), result, If(...))` with a single `BTreeMap<Row, usize>` lookup, turning O(n) evaluation into O(log n).
`CaseLiteral` replaces chains of `If(Eq(expr, literal), result, If(...))` with a sorted `Vec<CaseLiteralEntry>` + binary-search lookup, turning O(n) evaluation into O(log n).
Each `CaseLiteralEntry` pairs a literal `Row` with the index of the corresponding result expression in `exprs`.
Represented as `CallVariadic { func: CaseLiteral { lookup, return_type }, exprs }` where `exprs[0]` is the input expression, `exprs[1..n]` are case result expressions, and `exprs[last]` is the fallback.
NULL inputs go straight to the fallback since SQL `NULL = x` is always falsy.
Implements `LazyVariadicFunc` with `propagates_nulls = false`, `introduces_nulls = true`, `could_error = false`, `is_monotone = false`, `is_associative = false`.
Helper methods `get` (binary-search lookup returning an expr index) and `insert` (sorted insertion, overwriting duplicates and returning the old index) are provided for building and querying the lookup table.
4 changes: 2 additions & 2 deletions doc/developer/generated/expr/scalar/func/impls/jsonb.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
---
source: src/expr/src/scalar/func/impls/jsonb.rs
revision: af9155582e
revision: 90a38f32be
---

# mz-expr::scalar::func::impls::jsonb

Provides scalar function implementations for `jsonb` datums: subscript operators, `jsonb_array_elements`, `jsonb_each`, `jsonb_object_keys`, type-coercion casts, `jsonb_build_object`, `jsonb_build_array`, `jsonb_pretty`, and containment checks.
Also provides `parse_catalog_id` (converts catalog JSON-serialized IDs into string format like `u1`, `s2`, `p`) and `parse_catalog_privileges` (converts catalog JSON-serialized privilege arrays into `mz_aclitem[]`), with helper functions `jsonb_datum_to_u64` and `jsonb_datum_to_role_id` for the conversions.
Also provides `parse_catalog_id` (converts catalog JSON-serialized IDs into string format like `u1`, `s2`, `p`), `parse_catalog_privileges` (converts catalog JSON-serialized privilege arrays into `mz_aclitem[]`), and `parse_catalog_create_sql` (parses a catalog `create_sql` string into a JSONB object with type and, for materialized views, cluster and definition fields), with helper functions `jsonb_datum_to_u64` and `jsonb_datum_to_role_id` for the conversions.
4 changes: 2 additions & 2 deletions doc/developer/generated/expr/scalar/func/unary.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
---
source: src/expr/src/scalar/func/unary.rs
revision: af9155582e
revision: 90a38f32be
---

# mz-expr::scalar::func::unary

Defines the `LazyUnaryFunc` and `EagerUnaryFunc` traits, and uses `derive_unary\!` to generate the `UnaryFunc` enum.
`LazyUnaryFunc` is the core trait all unary scalar functions must implement (either directly or via the blanket impl from `EagerUnaryFunc`); it requires `eval`, `output_sql_type`, `propagates_nulls`, `introduces_nulls`, `preserves_uniqueness`, `inverse`, `is_monotone`, `is_eliminable_cast`, and optionally `could_error`.
`EagerUnaryFunc` is a higher-level trait using associated `Input`/`Output` types that implement `InputDatumType`/`OutputDatumType`; null propagation, null introduction, and error properties are inferred automatically from the type signatures. A blanket `impl<T: EagerUnaryFunc> LazyUnaryFunc for T` bridges the two traits.
The `derive_unary\!` invocation includes variants for all built-in unary functions, including `ParseCatalogId` and `ParseCatalogPrivileges` for catalog JSON deserialization.
The `derive_unary\!` invocation includes variants for all built-in unary functions, including `ParseCatalogId`, `ParseCatalogPrivileges`, and `ParseCatalogCreateSql` for catalog JSON deserialization, and `RedactSql` for SQL redaction.
These properties drive null-propagation, type inference, and optimizer rewrites throughout the system.
Loading
Loading