Skip to content

ACE-2020-add-timestamp_ntz-in-databricks-conn-config #43

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
anasY2 wants to merge 1 commit into
base: dev
Choose a base branch
from
Open

ACE-2020-add-timestamp_ntz-in-databricks-conn-config #43

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion content/docs/targets/target-setup/databricks/databricks-aws.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,14 @@ Replicant requires the Databricks JDBC Driver as a dependency. To obtain the app
max-connections: 30 #Maximum number of connections Replicant can open in Databricks
max-retries: 100 #Number of times any operation on the source system will be re-attempted on failures.
retry-wait-duration-ms: 1000 #Duration in milliseconds replicant should wait before performing then next retry of a failed operation

supports-timestamp-ntz: false #If disabled, uses timestamp data type instead of timestamp_ntz
date-format: <date format> #default is yyyy-MM-dd, specify the date format if source provides dates in a format other than default
timestamp-format: <timestamp format> #default 'yyyy-MM-dd HH:mm:ss', specify the format if source provides timestamps in a format other than default
```
- **`supports-timestamp-ntz`**: Specifies which datatype to use for timestamp data. `timestamp_ntz` stores timestamp without timezone.
- **`date-format`**: Specify the data format of the source database.
- **`timestamp-format`**: Specify the timestamp format of the source database.

Replace the following:
- *`HOSTNAME`*: the hostname of your Databricks host
Expand Down Expand Up @@ -302,4 +309,4 @@ As of now, note the following about the state of Arcion's Unity Catalog support:
*Default: `PARQUET`*.
- We use `SparkJDBC42` driver for Legacy Databricks (`DATABRICKS_DELTALAKE`) and `DatabricksJDBC42` for Unity catalog (`DATABRICKS_LAKEHOUSE`). For instructions on how to obtain these drivers, see [Obtain the JDBC Driver for Databricks](#i-obtain-the-jdbc-driver-for-databricks).
- Replicant supports Unity Catalog on AWS and Azure platforms.
- To configure [Mapper file]({{< ref "docs/targets/configuration-files/mapper-reference" >}}) in Unity Catalog, see [Mapping in Unity Catalog]({{< ref "docs/targets/configuration-files/mapper-reference#mapping-in-unity-catalog" >}}).
- To configure [Mapper file]({{< ref "docs/targets/configuration-files/mapper-reference" >}}) in Unity Catalog, see [Mapping in Unity Catalog]({{< ref "docs/targets/configuration-files/mapper-reference#mapping-in-unity-catalog" >}}).
10 changes: 9 additions & 1 deletion content/docs/targets/target-setup/databricks/databricks-azure.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,15 @@ In this step, you need to provide the Databricks connection details to Arcion. T
host: "HOSTNAME"
port: "PORT_NUMBER"
max-connections: 30

supports-timestamp-ntz: false #If disabled, uses timestamp data type instead of timestamp_ntz
date-format: <date format> #default is yyyy-MM-dd, specify the date format if source provides dates in a format other than default
timestamp-format: <timestamp format> #default 'yyyy-MM-dd HH:mm:ss', specify the format if source provides timestamps in a format other than default
```
- **`supports-timestamp-ntz`**: Specifies which datatype to use for timestamp data. `timestamp_ntz` stores timestamp without timezone.
- **`date-format`**: Specify the data format of the source database.
- **`timestamp-format`**: Specify the timestamp format of the source database.

Replace the following:
- *`JDBC_URL`*: the JDBC URL that you retrieve [from the connection details for a cluster](#get-connection-details-for-a-cluster)
- *`HOSTNAME`*: the hostname of your Databricks host that you retrieved [from the connection details for a cluster](#get-connection-details-for-a-cluster)
Expand Down Expand Up @@ -342,4 +350,4 @@ As of now, note the following about the state of Arcion's Unity Catalog support:
*Default: `PARQUET`*.
- We use `SparkJDBC42` driver for Legacy Databricks (`DATABRICKS_DELTALAKE`) and `DatabricksJDBC42` for Unity catalog (`DATABRICKS_LAKEHOUSE`). For instructions on how to obtain these drivers, see [Obtain the JDBC Driver for Databricks](#i-obtain-the-jdbc-driver-for-databricks).
- Replicant supports Unity Catalog on AWS and Azure platforms.
- To configure [Mapper file]({{< ref "docs/targets/configuration-files/mapper-reference" >}}) in Unity Catalog, see [Mapping in Unity Catalog]({{< ref "docs/targets/configuration-files/mapper-reference#mapping-in-unity-catalog" >}}).
- To configure [Mapper file]({{< ref "docs/targets/configuration-files/mapper-reference" >}}) in Unity Catalog, see [Mapping in Unity Catalog]({{< ref "docs/targets/configuration-files/mapper-reference#mapping-in-unity-catalog" >}}).