Dataform CLI run (dry) fails when trailing comments are in pre_operations

[spatel11]

Dataform core version 3.0.47

This definition will fail when running dataform run --dry-run

config {
  "type": "table",
}

SELECT 1 AS foo

pre_operations {

  DECLARE bar DATE DEFAULT DATE_TRUNC(CURRENT_DATE(), WEEK(MONDAY));

  -- DECLARE bar DATE DEFAULT DATE_TRUNC(DATE('2026-01-01'), WEEK(MONDAY))
}

Dataset creation failed: test.trailing_comment_test [table]
DECLARE bar DATE DEFAULT DATE_TRUNC(CURRENT_DATE(), WEEK(MONDAY));

-- DECLARE bar DATE DEFAULT DATE_TRUNC(DATE('2026-01-01'), WEEK(MONDAY))
;
create or replace table projectid.test.trailing_comment_test as

SELECT 1 AS foo
bigquery error: Syntax error: Unexpected ";" at [4:1]

It will also fail with:

DECLARE bar DATE DEFAULT DATE_TRUNC(CURRENT_DATE(), WEEK(MONDAY))

---

  -- DECLARE bar DATE DEFAULT DATE_TRUNC(DATE('2026-01-01'), WEEK(MONDAY))

Dataset creation failed: test.trailing_comment_test [table]
DECLARE bar DATE DEFAULT DATE_TRUNC(CURRENT_DATE(), WEEK(MONDAY))
;
-- DECLARE bar DATE DEFAULT DATE_TRUNC(DATE('2026-01-01'), WEEK(MONDAY))
;
create or replace table projectid.test.trailing_comment_test as

SELECT 1 AS foo
bigquery error: Syntax error: Unexpected ";" at [4:1]

both forms will work in the GCP dataform service (they'll compile and run).

Moving the comment above the declaration works.

[kolina]

I think the problem is in this line: we don't recognize that a SQL statement already has a semicolon due to a comment string.

Removing adding newline before semicolon would help to fix, but then it'll break an old issue.

I don't think that we can correctly handle both use cases without some level of understanding comments in SQL.

A workaround is to add semicolon to the end of comment string

[spatel11]

A workaround is to add semicolon to the end of comment string

I tried this, and it failed for the same reason. More interestingly, it does not show the ; as existing:

config {
  "type": "table",
}

SELECT
  1 AS foo

pre_operations {
  DECLARE
    bar DATE DEFAULT DATE_TRUNC(CURRENT_DATE(), WEEK(MONDAY));
    -- DECLARE bar DATE DEFAULT DATE_TRUNC(DATE('2026-01-01'), WEEK(MONDAY));
}

Dataset creation failed: test. trailing_comment_test [table]
DECLARE
bar DATE DEFAULT DATE_TRUNC(CURRENT_DATE(), WEEK(MONDAY));
-- DECLARE bar DATE DEFAULT DATE_TRUNC(DATE('2026-01-01'), WEEK(MONDAY))
;
create or replace table projectid.test.trailing_comment_test as

SELECT
1 AS foo
bigquery error: Syntax error: Unexpected ";" at [4:1]

[kolina]

I tried this, and it failed for the same reason. More interestingly, it does not show the ; as existing:

Yeah, you're right, it doesn't help because we strip a statement from ; if it's present and insert it after newline in the current logic.

The workaround can be fixed though if we change this implementation to not strip ; from the compiled statement and add \n; only if it doesn't already ends with ;. If you're ok with this workaround, feel free to send PR our way.

[spatel11]

I haven't forgotten about this. I'm currently trying to work out the CLA.