Problem
The ClickHouse IO currently supports only DateTime (second precision) via TypeName.DATETIME, which maps to Beam's Schema.FieldType.DATETIME (millisecond precision Joda Instant).
ClickHouse's DateTime64(precision, [timezone]) type — which supports millisecond, microsecond, and nanosecond precision — is not recognized by TableSchema or the column type parser, so users
cannot write sub-second timestamps to ClickHouse tables that use DateTime64 columns.
This is becoming a blocker for users whose pipelines emit events with sub-second timestamps (e.g. log/event ingestion, financial data).
Proposed solution
-
Schema model
- Add
TypeName.DATETIME64.
- Extend
ColumnType with a precision field (0–9) and an optional timezone string, plus a factory ColumnType.dateTime64(int precision, @Nullable String timezone).
-
Parser
- Add a grammar rule for
DateTime64(<precision>[, '<timezone>']) to ColumnTypeParser.jj.
-
Beam field type mapping
- For
precision <= 3, continue mapping to Schema.FieldType.DATETIME (Joda Instant, ms precision) for backward compatibility.
- For
precision > 3, map to a Beam logical type that preserves sub-second precision — SqlTypes.TIMESTAMP (java.time.Instant-backed, nanos) is the natural fit.
-
Writer
- In
ClickHouseWriter, serialize DateTime64 as a little-endian Int64 containing epoch_seconds * 10^precision + sub_second_units, matching the ClickHouse native protocol.
-
Tests
- Round-trip tests for precisions 3, 6, 9 against a ClickHouse test container, plus parser unit tests covering
DateTime64(3), DateTime64(6, 'UTC'), and the Nullable(DateTime64(...))
wrapper.
Issue Priority
Priority: 2 (default / most feature requests should be filed as P2)
Issue Components
Problem
The ClickHouse IO currently supports only
DateTime(second precision) viaTypeName.DATETIME, which maps to Beam'sSchema.FieldType.DATETIME(millisecond precision JodaInstant).ClickHouse's
DateTime64(precision, [timezone])type — which supports millisecond, microsecond, and nanosecond precision — is not recognized byTableSchemaor the column type parser, so userscannot write sub-second timestamps to ClickHouse tables that use
DateTime64columns.This is becoming a blocker for users whose pipelines emit events with sub-second timestamps (e.g. log/event ingestion, financial data).
Proposed solution
Schema model
TypeName.DATETIME64.ColumnTypewith aprecisionfield (0–9) and an optionaltimezonestring, plus a factoryColumnType.dateTime64(int precision, @Nullable String timezone).Parser
DateTime64(<precision>[, '<timezone>'])toColumnTypeParser.jj.Beam field type mapping
precision <= 3, continue mapping toSchema.FieldType.DATETIME(JodaInstant, ms precision) for backward compatibility.precision > 3, map to a Beam logical type that preserves sub-second precision —SqlTypes.TIMESTAMP(java.time.Instant-backed, nanos) is the natural fit.Writer
ClickHouseWriter, serializeDateTime64as a little-endianInt64containingepoch_seconds * 10^precision + sub_second_units, matching the ClickHouse native protocol.Tests
DateTime64(3),DateTime64(6, 'UTC'), and theNullable(DateTime64(...))wrapper.
Issue Priority
Priority: 2 (default / most feature requests should be filed as P2)
Issue Components