Skip to content

Add Databricks adapter to Delta file format configuration in elementary CLI #2154

@WillC3

Description

@WillC3

Hi - mostly LLM generated but expecting this to fix errors I'm seeing using elementary-data 0.23.0:

{"data": {"description": "sql incremental model elementary.alerts_v2", "execution_time": 0.048086643, "index": 1, "node_info": {"materialized": "incremental", "meta": {}, "node_finished_at": "2026-03-16T15:31:42.423947", "node_name": "alerts_v2", "node_path": "alerts/alerts_v2.sql", "node_relation": {"alias": "alerts_v2", "database": "xxx", "relation_name": "`xxx`.`elementary`.`alerts_v2`", "schema": "elementary"}, "node_started_at": "2026-03-16T15:31:42.374489", "node_status": "error", "resource_type": "model", "unique_id": "model.elementary_cli.alerts_v2"}, "status": "error", "total": 1}, "info": {"category": "", "code": "Q012", "extra": {}, "invocation_id": "11a3bf85-d57e-48f5-8b0a-23d218c8885b", "level": "error", "msg": "1 of 1 ERROR creating sql incremental model elementary.alerts_v2 ............... [\u001b[31mERROR\u001b[0m in 0.05s]", "name": "LogModelResult", "pid": 11, "thread": "Thread-2 (worker)", "ts": "2026-03-16T15:31:42.427791Z"}}
[2026-03-16, 15:31:43 UTC] {pod_manager.py:520} INFO - [base] {"data": {"msg": "Compilation Error in model alerts_v2 (models/alerts/alerts_v2.sql)\n  Invalid file format provided: None\n      Expected one of: text, csv, json, jdbc, parquet, orc, hive, delta, libsvm, hudi\n  \n  > in macro dbt_databricks_validate_get_file_format (macros/materializations/incremental/validate.sql)\n  > called by macro materialization_incremental_databricks (macros/materializations/incremental/incremental.sql)\n  > called by model alerts_v2 (models/alerts/alerts_v2.sql)", "node_info": {"materialized": "incremental", "meta": {}, "node_finished_at": "", "node_name": "alerts_v2", "node_path": "alerts/alerts_v2.sql", "node_relation": {"alias": "alerts_v2", "database": "xxx", "relation_name": "`xxx`.`elementary`.`alerts_v2`", "schema": "elementary"}, "node_started_at": "", "node_status": "None", "resource_type": "model", "unique_id": "model.elementary_cli.alerts_v2"}}, "info": {"category": "", "code": "Z024", "extra": {}, "invocation_id": "11a3bf85-d57e-48f5-8b0a-23d218c8885b", "level": "error", "msg": "  Compilation Error in model alerts_v2 (models/alerts/alerts_v2.sql)\n  Invalid file format provided: None\n      Expected one of: text, csv, json, jdbc, parquet, orc, hive, delta, libsvm, hudi\n  \n  > in macro dbt_databricks_validate_get_file_format (macros/materializations/incremental/validate.sql)\n  > called by macro materialization_incremental_databricks (macros/materializations/incremental/incremental.sql)\n  > called by model alerts_v2 (models/alerts/alerts_v2.sql)", "name": "RunResultError", "pid": 11, "thread": "MainThread", "ts": "2026-03-16T15:31:42.693318Z"}}
[2026-03-16, 15:31:43 UTC] {pod_manager.py:520} INFO - [base] 

Is your feature request related to a problem? Please describe.

When using Elementary CLI with Databricks, the alerts_v2 incremental model fails to use Delta file format because 'databricks' is not included in the adapter type check. Currently, the configuration only checks for ['spark', 'fabricspark']:

models:
  elementary_cli:
    +file_format: "{{ 'delta' if target.type in ['spark', 'fabricspark'] else none }}"

This causes issues because:

  • Databricks requires Delta format for proper incremental model operation with ACID transactions
  • The alerts_v2 model uses incremental materialization with merge operations that need Delta support
  • Without Delta format, the monitor alerts functionality may not work correctly

Describe the solution you'd like

Add 'databricks' to the list of adapter types that should use Delta file format:

models:
  elementary_cli:
    +file_format: "{{ 'delta' if target.type in ['spark', 'fabricspark', 'databricks'] else none }}"

File location: dbt_project.yml (line 40)

Describe alternatives you've considered

  • Forking the repository and maintaining a custom version
  • Attempting to override via user dbt_project.yml (doesn't work since CLI uses its own bundled config)
  • Using Spark adapter instead of Databricks adapter (not ideal for Databricks-specific features)

Additional context

Similar support was recently added for other adapters:

This appears to be a small oversight since Databricks has extensive support elsewhere in the Elementary codebase.

Would you be willing to contribute this feature?

Yes! I have the one-line change ready and tested locally. Happy to submit a PR if given contributor access, or can provide the specific change for a maintainer to apply.


Would you like me to help you submit this issue on GitHub? (You'll need to copy/paste it since I can't directly create GitHub issues)

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions