Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 39 additions & 37 deletions data/integrations.yml
Original file line number Diff line number Diff line change
Expand Up @@ -256,66 +256,68 @@ integrations:
# Databases
#####

- name: Hive
icon: ":simple-apachehive:"
content: |
Read from or write to an embedded Apache {{p.Hive}} endpoint.

- name: pgvector
icon: ":black_large_square:"
content: |
Store vector embeddings into [pgvector](https://github.com/pgvector/pgvector)
using the {{p.cmem_plugin_pgvector_Search}}.

- name: Neo4J
icon: ":simple-neo4j:"
content: |
Use the {{p.neo4j}} dataset for reading and writing [Neo4j graphs](https://neo4j.com/).

- name: PostgreSQL
icon: ":simple-postgresql:"
- name: Snowflake
icon: ":simple-snowflake:"
content: |
PostgreSQL can be accessed with the {{p.Jdbc}} dataset and a
[JDBC driver](https://central.sonatype.com/artifact/org.postgresql/postgresql/versions).
Snowflake can be accessed with the {{p.SnowflakeJdbc}} dataset (JDBC driver included).

- name: MariaDB
icon: ":simple-mariadb:"
- name: Microsoft SQL
icon: ":material-microsoft:"
content: |
MariaDB can be accessed with the {{p.Jdbc}} dataset and a
[JDBC driver](https://central.sonatype.com/artifact/org.mariadb.jdbc/mariadb-java-client/overview).
The Microsoft SQL Server can be accessed with the {{p.Jdbc}} dataset (JDBC driver included).

- name: SQLite
icon: ":simple-sqlite:"
- name: PostgreSQL
icon: ":simple-postgresql:"
content: |
SQLite can be accessed with the {{p.Jdbc}} dataset and a
[JDBC driver](https://central.sonatype.com/artifact/org.xerial/sqlite-jdbc).
PostgreSQL can be accessed with the {{p.Jdbc}} dataset (JDBC driver included).

- name: MySQL
icon: ":simple-mysql:"
content: |
MySQL can be accessed with the {{p.Jdbc}} dataset and a
[JDBC driver](https://central.sonatype.com/artifact/org.mariadb.jdbc/mariadb-java-client/overview).

- name: Hive
icon: ":simple-apachehive:"
content: |
Read from or write to an embedded Apache {{p.Hive}} endpoint.
MySQL can be accessed with the {{p.Jdbc}} dataset (JDBC driver included).

- name: Microsoft SQL
icon: ":material-microsoft:"
content: |
The Microsoft SQL Server can be accessed with the {{p.Jdbc}} dataset and a
[JDBC driver](https://central.sonatype.com/artifact/com.microsoft.sqlserver/mssql-jdbc).

- name: Snowflake
icon: ":simple-snowflake:"
- name: MariaDB
icon: ":simple-mariadb:"
content: |
Snowflake can be accessed with the {{p.SnowflakeJdbc}} dataset and a
[JDBC driver](https://central.sonatype.com/artifact/net.snowflake/snowflake-jdbc).
MariaDB can be accessed with the {{p.Jdbc}} dataset (JDBC driver included).

- name: pgvector
icon: ":black_large_square:"
- name: SQLite
icon: ":simple-sqlite:"
content: |
Store vector embeddings into [pgvector](https://github.com/pgvector/pgvector)
using the {{p.cmem_plugin_pgvector_Search}}.
SQLite can be accessed with the {{p.Jdbc}} dataset and a
[Custom JDBC driver](https://central.sonatype.com/artifact/org.xerial/sqlite-jdbc).
Please have a look at
[Setup and use of JDBC Drivers](../../deploy-and-configure/configuration/dataintegration/jdbc/).

- name: Trino
icon: ":simple-trino:"
content: |
[Trino](https://github.com/trinodb/trino) can be access with the
{{p.Jdbc}} dataset and a [JDBC driver](https://trino.io/docs/current/client/jdbc.html).
[Trino](https://github.com/trinodb/trino) can be access with the {{p.Jdbc}} dataset and a
[Custom JDBC driver](https://trino.io/docs/current/client/jdbc.html).
Please have a look at
[Setup and use of JDBC Drivers](../../deploy-and-configure/configuration/dataintegration/jdbc/).

- name: Databricks
icon: ":simple-databricks:"
content: |
[Databricks](https://www.databricks.com) can be access with the
{{p.Jdbc}} dataset and a [JDBC driver](https://www.databricks.com/spark/jdbc-drivers-download).
[Databricks](http://databricks.com/) can be accessed with the {{p.Jdbc}} dataset and a
[Custom JDBC driver](https://github.com/databricks/databricks-jdbc).
Please have a look at
[Setup and use of JDBC Drivers](../../deploy-and-configure/configuration/dataintegration/jdbc/).

31 changes: 16 additions & 15 deletions docs/build/integrations/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,8 +51,10 @@ to interact with any [Azure AI Foundry provided Large Language Models](https://a

---

[Databricks](https://www.databricks.com) can be access with the
[Remote SQL endpoint](../../build/reference/dataset/Jdbc.md) dataset and a [JDBC driver](https://www.databricks.com/spark/jdbc-drivers-download).
[Databricks](http://databricks.com/) can be accessed with the [Remote SQL endpoint](../../build/reference/dataset/Jdbc.md) dataset and a
[Custom JDBC driver](https://github.com/databricks/databricks-jdbc).
Please have a look at
[Setup and use of JDBC Drivers](../../deploy-and-configure/configuration/dataintegration/jdbc/).


- :material-email-outline:{ .lg .middle } eMail / SMTP
Expand Down Expand Up @@ -142,8 +144,7 @@ GraphDB can be used as the integrated Quad Store as well.

---

MariaDB can be accessed with the [Remote SQL endpoint](../../build/reference/dataset/Jdbc.md) dataset and a
[JDBC driver](https://central.sonatype.com/artifact/org.mariadb.jdbc/mariadb-java-client/overview).
MariaDB can be accessed with the [Remote SQL endpoint](../../build/reference/dataset/Jdbc.md) dataset (JDBC driver included).


- :simple-mattermost:{ .lg .middle } Mattermost
Expand All @@ -158,16 +159,14 @@ the [Send Mattermost messages](../../build/reference/customtask/cmem_plugin_matt

---

The Microsoft SQL Server can be accessed with the [Remote SQL endpoint](../../build/reference/dataset/Jdbc.md) dataset and a
[JDBC driver](https://central.sonatype.com/artifact/com.microsoft.sqlserver/mssql-jdbc).
The Microsoft SQL Server can be accessed with the [Remote SQL endpoint](../../build/reference/dataset/Jdbc.md) dataset (JDBC driver included).


- :simple-mysql:{ .lg .middle } MySQL

---

MySQL can be accessed with the [Remote SQL endpoint](../../build/reference/dataset/Jdbc.md) dataset and a
[JDBC driver](https://central.sonatype.com/artifact/org.mariadb.jdbc/mariadb-java-client/overview).
MySQL can be accessed with the [Remote SQL endpoint](../../build/reference/dataset/Jdbc.md) dataset (JDBC driver included).


- :simple-neo4j:{ .lg .middle } Neo4J
Expand Down Expand Up @@ -254,8 +253,7 @@ using the [Search Vector Embeddings](../../build/reference/customtask/cmem_plugi

---

PostgreSQL can be accessed with the [Remote SQL endpoint](../../build/reference/dataset/Jdbc.md) dataset and a
[JDBC driver](https://central.sonatype.com/artifact/org.postgresql/postgresql/versions).
PostgreSQL can be accessed with the [Remote SQL endpoint](../../build/reference/dataset/Jdbc.md) dataset (JDBC driver included).


- :other-powerbi:{ .lg .middle } PowerBI
Expand Down Expand Up @@ -315,8 +313,7 @@ execute a [SOQL query (Salesforce)](../../build/reference/customtask/cmem_plugin

---

Snowflake can be accessed with the [Snowflake SQL endpoint](../../build/reference/dataset/SnowflakeJdbc.md) dataset and a
[JDBC driver](https://central.sonatype.com/artifact/net.snowflake/snowflake-jdbc).
Snowflake can be accessed with the [Snowflake SQL endpoint](../../build/reference/dataset/SnowflakeJdbc.md) dataset (JDBC driver included).


- :simple-apachespark:{ .lg .middle } Spark
Expand All @@ -331,7 +328,9 @@ execute a [SOQL query (Salesforce)](../../build/reference/customtask/cmem_plugin
---

SQLite can be accessed with the [Remote SQL endpoint](../../build/reference/dataset/Jdbc.md) dataset and a
[JDBC driver](https://central.sonatype.com/artifact/org.xerial/sqlite-jdbc).
[Custom JDBC driver](https://central.sonatype.com/artifact/org.xerial/sqlite-jdbc).
Please have a look at
[Setup and use of JDBC Drivers](../../deploy-and-configure/configuration/dataintegration/jdbc/).


- :material-ssh:{ .lg .middle } SSH
Expand All @@ -357,8 +356,10 @@ Tentris can be used as the integrated Quad Store as well (beta).

---

[Trino](https://github.com/trinodb/trino) can be access with the
[Remote SQL endpoint](../../build/reference/dataset/Jdbc.md) dataset and a [JDBC driver](https://trino.io/docs/current/client/jdbc.html).
[Trino](https://github.com/trinodb/trino) can be access with the [Remote SQL endpoint](../../build/reference/dataset/Jdbc.md) dataset and a
[Custom JDBC driver](https://trino.io/docs/current/client/jdbc.html).
Please have a look at
[Setup and use of JDBC Drivers](../../deploy-and-configure/configuration/dataintegration/jdbc/).


- :black_large_square:{ .lg .middle } Virtuoso
Expand Down
1 change: 1 addition & 0 deletions docs/build/loading-jdbc-datasets-incrementally/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
icon: material/database-sync
tags:
- ExpertTutorial
- JDBC
---
# Loading JDBC datasets incrementally

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
nav:
- Build (DataIntegration): index.md
- Adding JDBC drivers: jdbc
- Activity Reference: activity-reference

Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
---
tags:
- Configuration
- JDBC
---
# Setup and use of JDBC Drivers

Corporate Memory supports JDBC connections to database management systems (DBMSs).
The platform includes several JDBC drivers by default.
You can also add and use custom drivers.

!!! info "References"

For more technical details, see the following reference pages:

- [Remote SQL endpoint](../../../../build/reference/dataset/Jdbc.md) and
- [Snowflake SQL endpoint](../../../../build/reference/dataset/SnowflakeJdbc.md).

## Bundled JDBC Drivers

The platform includes the following JDBC drivers:

- PostgreSQL (`postgresql v42.7.10`)
- MariaDB (includes support for MySQL, `mariadb-java-client v3.5.7`)
- Microsoft SQL Server (`mssql-jdbc v13.2.1.jre11`)
- Snowflake (`snowflake-jdbc v3.28.0`)

## Custom JDBC Drivers

In addition to the bundled JDBC drivers, you can register custom JDBC drivers.
The following sections describe the required configuration.

### Download Custom JDBC Driver

Download the JDBC driver for each database management system that you want to connect to.
[Integrations](../../../../build/integrations/index.md) provides links for well-known systems and lists those that are actively used with Corporate Memory.

### Provide a Custom JDBC Driver

Consult your solutions manager or DevOps specialist for options to copy or inject the JDBC driver `jar` into a Corporate Memory deployment.
Depending on the deployment model, suitable options include:

- The Docker Compose package `cmem-orchestration` mounts the folder `./conf/dataintegration/plugin/` into the DataIntegration container.
The configuration snippets below assume this location, which maps to `/opt/cmem/eccenca-DataIntegration/dist/etc/dataintegration/conf/plugin/` inside the container.
- A dedicated _Build project_ in which the driver JAR files are uploaded as project file resources.
- Dedicated file or resource mounts in a Docker Compose or Helm/Kubernetes configuration.

## Driver Registration

A custom JDBC driver must be registered in the DataIntegration configuration file `dataintegration.conf`, in the `spark.sql.options` section.
The following example shows how to register a custom JDBC driver for Databricks:

```conf
spark.sql.options {
# driver name
jdbc.drivers = "databricks"
# path to the jar in the docker container
jdbc.databricks.jar = "/opt/cmem/eccenca-DataIntegration/dist/etc/dataintegration/conf/plugin/DatabricksJDBC.jar"
# class name
jdbc.databricks.name = "com.databricks.client.jdbc.Driver"
}
```

## Use the Driver

JDBC drivers are used through the **Remote SQL endpoint** or **Snowflake SQL endpoint** dataset type.

![](jdbc-dataset.png){ class="bordered" width="85%" }

Configure them in the dataset configuration dialog.
For details about the JDBC connection string, consult your DBMS or JDBC driver documentation.

![](jdbc-config-databricks.png){ class="bordered" width="100%" }
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 2 additions & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@ extra:
Volume: fontawesome-solid-hard-drive
"Load Balancer": simple-awselasticloadbalancing
Variables: variables
JDBC: jdbc
# https://squidfunk.github.io/mkdocs-material/setup/setting-up-versioning/
version:
provider: mike
Expand Down Expand Up @@ -186,6 +187,7 @@ theme:
simple-awselasticloadbalancing: simple/awselasticloadbalancing
fontawesome-solid-hard-drive: fontawesome/solid/hard-drive
variables: material/variable-box
jdbc: material/database
# https://squidfunk.github.io/mkdocs-material/reference/annotations/
admonition:
note: fontawesome/solid/note-sticky
Expand Down
Loading