-
Notifications
You must be signed in to change notification settings - Fork 210
Description
Hi - mostly LLM generated but expecting this to fix errors I'm seeing using elementary-data 0.23.0:
{"data": {"description": "sql incremental model elementary.alerts_v2", "execution_time": 0.048086643, "index": 1, "node_info": {"materialized": "incremental", "meta": {}, "node_finished_at": "2026-03-16T15:31:42.423947", "node_name": "alerts_v2", "node_path": "alerts/alerts_v2.sql", "node_relation": {"alias": "alerts_v2", "database": "xxx", "relation_name": "`xxx`.`elementary`.`alerts_v2`", "schema": "elementary"}, "node_started_at": "2026-03-16T15:31:42.374489", "node_status": "error", "resource_type": "model", "unique_id": "model.elementary_cli.alerts_v2"}, "status": "error", "total": 1}, "info": {"category": "", "code": "Q012", "extra": {}, "invocation_id": "11a3bf85-d57e-48f5-8b0a-23d218c8885b", "level": "error", "msg": "1 of 1 ERROR creating sql incremental model elementary.alerts_v2 ............... [\u001b[31mERROR\u001b[0m in 0.05s]", "name": "LogModelResult", "pid": 11, "thread": "Thread-2 (worker)", "ts": "2026-03-16T15:31:42.427791Z"}}
[2026-03-16, 15:31:43 UTC] {pod_manager.py:520} INFO - [base] {"data": {"msg": "Compilation Error in model alerts_v2 (models/alerts/alerts_v2.sql)\n Invalid file format provided: None\n Expected one of: text, csv, json, jdbc, parquet, orc, hive, delta, libsvm, hudi\n \n > in macro dbt_databricks_validate_get_file_format (macros/materializations/incremental/validate.sql)\n > called by macro materialization_incremental_databricks (macros/materializations/incremental/incremental.sql)\n > called by model alerts_v2 (models/alerts/alerts_v2.sql)", "node_info": {"materialized": "incremental", "meta": {}, "node_finished_at": "", "node_name": "alerts_v2", "node_path": "alerts/alerts_v2.sql", "node_relation": {"alias": "alerts_v2", "database": "xxx", "relation_name": "`xxx`.`elementary`.`alerts_v2`", "schema": "elementary"}, "node_started_at": "", "node_status": "None", "resource_type": "model", "unique_id": "model.elementary_cli.alerts_v2"}}, "info": {"category": "", "code": "Z024", "extra": {}, "invocation_id": "11a3bf85-d57e-48f5-8b0a-23d218c8885b", "level": "error", "msg": " Compilation Error in model alerts_v2 (models/alerts/alerts_v2.sql)\n Invalid file format provided: None\n Expected one of: text, csv, json, jdbc, parquet, orc, hive, delta, libsvm, hudi\n \n > in macro dbt_databricks_validate_get_file_format (macros/materializations/incremental/validate.sql)\n > called by macro materialization_incremental_databricks (macros/materializations/incremental/incremental.sql)\n > called by model alerts_v2 (models/alerts/alerts_v2.sql)", "name": "RunResultError", "pid": 11, "thread": "MainThread", "ts": "2026-03-16T15:31:42.693318Z"}}
[2026-03-16, 15:31:43 UTC] {pod_manager.py:520} INFO - [base]
Is your feature request related to a problem? Please describe.
When using Elementary CLI with Databricks, the alerts_v2 incremental model fails to use Delta file format because 'databricks' is not included in the adapter type check. Currently, the configuration only checks for ['spark', 'fabricspark']:
models:
elementary_cli:
+file_format: "{{ 'delta' if target.type in ['spark', 'fabricspark'] else none }}"This causes issues because:
- Databricks requires Delta format for proper incremental model operation with ACID transactions
- The
alerts_v2model uses incremental materialization with merge operations that need Delta support - Without Delta format, the monitor alerts functionality may not work correctly
Describe the solution you'd like
Add 'databricks' to the list of adapter types that should use Delta file format:
models:
elementary_cli:
+file_format: "{{ 'delta' if target.type in ['spark', 'fabricspark', 'databricks'] else none }}"File location: dbt_project.yml (line 40)
Describe alternatives you've considered
- Forking the repository and maintaining a custom version
- Attempting to override via user dbt_project.yml (doesn't work since CLI uses its own bundled config)
- Using Spark adapter instead of Databricks adapter (not ideal for Databricks-specific features)
Additional context
Similar support was recently added for other adapters:
- feat: add dbt-fabricspark support to elementary CLI #2142 - fabricspark support
- feat: add Microsoft Fabric and SQL Server support to elementary CLI #2140 - Microsoft Fabric support
- feat: add DuckDB, Trino, Dremio & Spark support to CI and CLI #2135 - DuckDB, Trino, Dremio & Spark support
This appears to be a small oversight since Databricks has extensive support elsewhere in the Elementary codebase.
Would you be willing to contribute this feature?
Yes! I have the one-line change ready and tested locally. Happy to submit a PR if given contributor access, or can provide the specific change for a maintainer to apply.
Would you like me to help you submit this issue on GitHub? (You'll need to copy/paste it since I can't directly create GitHub issues)